From patchwork Fri Jun 9 01:06:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273061 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A47D9C7EE23 for ; Fri, 9 Jun 2023 01:07:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0BF9D8E000F; Thu, 8 Jun 2023 21:07:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0705A8E0001; Thu, 8 Jun 2023 21:07:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E7A518E000F; Thu, 8 Jun 2023 21:07:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id DBB298E0001 for ; Thu, 8 Jun 2023 21:07:06 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id A4FE41A0390 for ; Fri, 9 Jun 2023 01:07:06 +0000 (UTC) X-FDA: 80881420452.21.FF943C0 Received: from mail-yw1-f180.google.com (mail-yw1-f180.google.com [209.85.128.180]) by imf02.hostedemail.com (Postfix) with ESMTP id BA80980010 for ; Fri, 9 Jun 2023 01:07:04 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=F4iPSaqG; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of hughd@google.com designates 209.85.128.180 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686272824; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FYWG+Dmea1iVO9u1anz9ea+vH7b9XYdNpJnPYqrcQ8g=; b=hT4ORVWF2FKjypi4w2K+bfmiSwYNiLFjMOVRe0v9lqLGvGuRJKHt7ATXMCARvu8I44mGub UKRSMwoHCiec71Ig3bIRNMrJkG4BJF4M1gaG47Q5DBYf0Uj30J6ElQNB3EESbt0JFZtzOJ 69vmRMRamxEe8NRsV5xls5OGxrMDvQU= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=F4iPSaqG; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of hughd@google.com designates 209.85.128.180 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686272824; a=rsa-sha256; cv=none; b=t2mBGTptZdlkXcYWg068YHPuj+wzAnjVEnbd9O7nXGLEPyS6xHvYu4StIkWowMl+cZsiVi GMlk/Sp/LAQq6qlCuIUTA+fs5MP5+X0yWiX4X1zavMKCpr9liLAkB0X7dDj05V4vCgs7YM UU1KlmORQ7srRdHy9Qo7MwS7MHs2TD8= Received: by mail-yw1-f180.google.com with SMTP id 00721157ae682-568ba7abc11so11400847b3.3 for ; Thu, 08 Jun 2023 18:07:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686272824; x=1688864824; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=FYWG+Dmea1iVO9u1anz9ea+vH7b9XYdNpJnPYqrcQ8g=; b=F4iPSaqGVauSHCN3PPx1HPCWJrbnjh20XWi+31Rxj6kybI3XEkrDo9g8TlN/DUHnID cA24qUKu4G6Vj7NJbRdZASctuqFdhdFDB+bQFh7yqdfrUkggC5zThxCrXeltsHoErZDt xA3Doa45fnOu6mBH+w4Zu0i9kibQ9g3niiOe6jmkqXtnnKBsq4Rlqd1i923q7ctcGoK9 NpGckUH03ViLvNgz5x9NiKAf8htS6SXGATYEGV3wZ9h6z7DQULhRQ42RGKscWpa7yH/X UrzgAwS5xf827WDV7tLvd4mt4ZGJ4FS3qwkbBLtNfkKh0VkQABmVM7ywrrTk/wDqi4Mi rRxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686272824; x=1688864824; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=FYWG+Dmea1iVO9u1anz9ea+vH7b9XYdNpJnPYqrcQ8g=; b=JN9HWqwu+Pd4VNkI2S6nZWgOayb6VQlJnGg98BAuC1R9TaPOzRG8FkjMeUzugeDhEi /VBcjw86k+aL62F/mMhOBixt+qDAUbH5omhcEqO6627ZldjRN+9wYROVLL29VUN4WLt8 kAwsV4wsRFsEqjHR7kzB3sb1hjawQmJ/2iWKs7pnXsPQXz74p/XgGglAOJA8Yt+ixV+E bFySLwDs48QVCzySzVyYYtoWGMK+f8TBmRQtxvD0Y0j+8Lp9btYuhc1QiksDkCwmHJd7 OGhBYRhYimksKVk/1nljauH6bso1peaMzfYjeqE7ze6T4wUxHPJDmFnRv+fHp38bEnJ0 iNFg== X-Gm-Message-State: AC+VfDxXLMwzaqZX+EuA6+iPzwqmFLLN+Y7Tn+AYCTT+cqqrhfUezKmr hmqq1ppGeRpKhMZ03KhBmY/lxA== X-Google-Smtp-Source: ACHHUZ4e9gtItKHo/uKH6fckN0QfmcYW2cVTmMNzekmPXGojmR2zjeAFUImej/93RTPFo775/xnN5A== X-Received: by 2002:a81:7b0b:0:b0:561:afca:5b4d with SMTP id w11-20020a817b0b000000b00561afca5b4dmr1387844ywc.3.1686272823636; Thu, 08 Jun 2023 18:07:03 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id y4-20020a81a104000000b00566e949fb9esm282673ywg.82.2023.06.08.18.07.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:07:02 -0700 (PDT) Date: Thu, 8 Jun 2023 18:06:53 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 01/32] mm: use pmdp_get_lockless() without surplus barrier() In-Reply-To: Message-ID: References: MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: BA80980010 X-Stat-Signature: k3o8of3ajqefh6qzyjp5i5tcmbxcfaea X-Rspam-User: X-HE-Tag: 1686272824-821239 X-HE-Meta: U2FsdGVkX1+H4k5+xSZdSEypJ8tTCJmwG7+lq+XinTEAFlSIPsUnIqbZi+GNd5MzfjtaaTdC2o7HlZ+A5luwjIncA5BKMXypi/xvyFwo6FnLLwsZcCvBh26pDKgza6mJeYQvJMCslMwgBG8o+3dgfw3YiDcfVjvmUU0XuWmFpiYKnTSntqWAN464MOu2zW2lJHnn/qxTnYEKaWISKeMiuz/36fk5DQy1h0Cr/Xx37+pBI24cL5jGcsBZy9F9Ph+TKUWjgDbmzg0obfmxSWMR+8i1c5ocY7eiC5aqUMIHXA23uGTMILKVDB2fuEq9DUx8vagI26hmeTdqPnX82Gzz7U2LrK8ZBbX2EvsSpuGt616QIlwEhBgRBDz6Mjf/YBUTA6KrqMPo7gHQx7sTOkoTiLxHvnXblUFYadPwTIBU+yHuiYGRJeQGooYE+FX0ZnG82eDBEDTX1nnPkc0fD/4yvHsx2LaGDDn9wLvdUNRY2kuIRXla2+6zTsXIWqt8zTuYc+OSDN2SuyLbU+16DywWOZUOlcoINQmtG+i4EX0VcGT2CqU+ixOc6qyyjvxFg7vy1xE35ckT0Y75JFxqYwY8zRb/67ZnOFfZyJ1zu8/+6PkzzE57Ya1oi7yHo6uZ1gslQJexTJ9B68spOVFm4/31jzMMcpVZol/7kp4kJC+xyvSqE+MzQD5vTMcvuLzjwKVy2Ti0/H179efAOzXn0BiWXg3CJEuubSaquX2t810tDypUYbf4IUG1QXnT3gKdqJa2SeNYbsBcPp4KivPI9ObNoDfr7Is3x9pV3tgY28qSROI4/apn+GDkNNIjsRflioDtoz/Oz3UonYlnQe05iqfYJj71X3MlWxUonNKzF3resUiEoMlxMaQeDksgQoNEdvCMnd8bccZkUmu2eQs00V7Ux2e8XpDlIaMSWVl85GESF4R7FjhKCqHHoiv8Ed4yfkmkZgi1/bynD0YYDuoEWJX 8EJwJ9ze mKXghBa9xHHYdqFW3IWddfpNgNDjX29NyUD0gRegUDzac0WcdDTxB6Ceh7Q1azv+VhVV7djQmse/7JAVn9mBCN6mz4f9kCM3ND6ui8if0ob13UeaEj/d2PwJz82q/HGSccGIQLivqQ1LZzBkf/zZHMLNEXY3WwzuYNycjX0gWc7UUMTpCHc+N9ykiOxWJNtZ+SNdJOaLOpn76re0kuTX1bxlsz4eQls42kHSYJDs3BsvgYrYS0x0Gj319pwy/PeEVIyARkZ5Rk3ojyt3qsmv8tWloFVwWiwRx61ofhkplTmYQQbHvSSzPvxs3XvYx9EXwclG5qgtHKdEN4pzHMUe2NwZM8ePmAcofhKd3jye2ohedr+iKcfeNmVvb3K/Zvyck9L76GZqfm3EZnVTiHv6bNvlMDRVhsk7GUh/+j6AOk8e3NdhoxHjxRoYU67/X6hffDRhmuHi0hB1rWGc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use pmdp_get_lockless() in preference to READ_ONCE(*pmdp), to get a more reliable result with PAE (or READ_ONCE as before without PAE); and remove the unnecessary extra barrier()s which got left behind in its callers. HOWEVER: Note the small print in linux/pgtable.h, where it was designed specifically for fast GUP, and depends on interrupts being disabled for its full guarantee: most callers which have been added (here and before) do NOT have interrupts disabled, so there is still some need for caution. Signed-off-by: Hugh Dickins Acked-by: Yu Zhao Acked-by: Peter Xu --- fs/userfaultfd.c | 10 +--------- include/linux/pgtable.h | 17 ----------------- mm/gup.c | 6 +----- mm/hmm.c | 2 +- mm/khugepaged.c | 5 ----- mm/ksm.c | 3 +-- mm/memory.c | 14 ++------------ mm/mprotect.c | 5 ----- mm/page_vma_mapped.c | 2 +- 9 files changed, 7 insertions(+), 57 deletions(-) diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 0fd96d6e39ce..f7a0817b1ec0 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -349,15 +349,7 @@ static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx, if (!pud_present(*pud)) goto out; pmd = pmd_offset(pud, address); - /* - * READ_ONCE must function as a barrier with narrower scope - * and it must be equivalent to: - * _pmd = *pmd; barrier(); - * - * This is to deal with the instability (as in - * pmd_trans_unstable) of the pmd. - */ - _pmd = READ_ONCE(*pmd); + _pmd = pmdp_get_lockless(pmd); if (pmd_none(_pmd)) goto out; diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index c5a51481bbb9..8ec27fe69dc8 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1344,23 +1344,6 @@ static inline int pud_trans_unstable(pud_t *pud) static inline int pmd_none_or_trans_huge_or_clear_bad(pmd_t *pmd) { pmd_t pmdval = pmdp_get_lockless(pmd); - /* - * The barrier will stabilize the pmdval in a register or on - * the stack so that it will stop changing under the code. - * - * When CONFIG_TRANSPARENT_HUGEPAGE=y on x86 32bit PAE, - * pmdp_get_lockless is allowed to return a not atomic pmdval - * (for example pointing to an hugepage that has never been - * mapped in the pmd). The below checks will only care about - * the low part of the pmd with 32bit PAE x86 anyway, with the - * exception of pmd_none(). So the important thing is that if - * the low part of the pmd is found null, the high part will - * be also null or the pmd_none() check below would be - * confused. - */ -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - barrier(); -#endif /* * !pmd_present() checks for pmd migration entries * diff --git a/mm/gup.c b/mm/gup.c index bbe416236593..3bd5d3854c51 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -653,11 +653,7 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, struct mm_struct *mm = vma->vm_mm; pmd = pmd_offset(pudp, address); - /* - * The READ_ONCE() will stabilize the pmdval in a register or - * on the stack so that it will stop changing under the code. - */ - pmdval = READ_ONCE(*pmd); + pmdval = pmdp_get_lockless(pmd); if (pmd_none(pmdval)) return no_page_table(vma, flags); if (!pmd_present(pmdval)) diff --git a/mm/hmm.c b/mm/hmm.c index 6a151c09de5e..e23043345615 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -332,7 +332,7 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, pmd_t pmd; again: - pmd = READ_ONCE(*pmdp); + pmd = pmdp_get_lockless(pmdp); if (pmd_none(pmd)) return hmm_vma_walk_hole(start, end, -1, walk); diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 6b9d39d65b73..732f9ac393fc 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -961,11 +961,6 @@ static int find_pmd_or_thp_or_none(struct mm_struct *mm, return SCAN_PMD_NULL; pmde = pmdp_get_lockless(*pmd); - -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - /* See comments in pmd_none_or_trans_huge_or_clear_bad() */ - barrier(); -#endif if (pmd_none(pmde)) return SCAN_PMD_NONE; if (!pmd_present(pmde)) diff --git a/mm/ksm.c b/mm/ksm.c index 0156bded3a66..df2aa281d49d 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -1194,8 +1194,7 @@ static int replace_page(struct vm_area_struct *vma, struct page *page, * without holding anon_vma lock for write. So when looking for a * genuine pmde (in which to find pte), test present and !THP together. */ - pmde = *pmd; - barrier(); + pmde = pmdp_get_lockless(pmd); if (!pmd_present(pmde) || pmd_trans_huge(pmde)) goto out; diff --git a/mm/memory.c b/mm/memory.c index f69fbc251198..2eb54c0d5d3c 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4925,18 +4925,9 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) * So now it's safe to run pte_offset_map(). */ vmf->pte = pte_offset_map(vmf->pmd, vmf->address); - vmf->orig_pte = *vmf->pte; + vmf->orig_pte = ptep_get_lockless(vmf->pte); vmf->flags |= FAULT_FLAG_ORIG_PTE_VALID; - /* - * some architectures can have larger ptes than wordsize, - * e.g.ppc44x-defconfig has CONFIG_PTE_64BIT=y and - * CONFIG_32BIT=y, so READ_ONCE cannot guarantee atomic - * accesses. The code below just needs a consistent view - * for the ifs and we later double check anyway with the - * ptl lock held. So here a barrier will do. - */ - barrier(); if (pte_none(vmf->orig_pte)) { pte_unmap(vmf->pte); vmf->pte = NULL; @@ -5060,9 +5051,8 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, if (!(ret & VM_FAULT_FALLBACK)) return ret; } else { - vmf.orig_pmd = *vmf.pmd; + vmf.orig_pmd = pmdp_get_lockless(vmf.pmd); - barrier(); if (unlikely(is_swap_pmd(vmf.orig_pmd))) { VM_BUG_ON(thp_migration_supported() && !is_pmd_migration_entry(vmf.orig_pmd)); diff --git a/mm/mprotect.c b/mm/mprotect.c index 92d3d3ca390a..c5a13c0f1017 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -309,11 +309,6 @@ static inline int pmd_none_or_clear_bad_unless_trans_huge(pmd_t *pmd) { pmd_t pmdval = pmdp_get_lockless(pmd); - /* See pmd_none_or_trans_huge_or_clear_bad for info on barrier */ -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - barrier(); -#endif - if (pmd_none(pmdval)) return 1; if (pmd_trans_huge(pmdval)) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 4e448cfbc6ef..64aff6718bdb 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -210,7 +210,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) * compiler and used as a stale value after we've observed a * subsequent update. */ - pmde = READ_ONCE(*pvmw->pmd); + pmde = pmdp_get_lockless(pvmw->pmd); if (pmd_trans_huge(pmde) || is_pmd_migration_entry(pmde) || (pmd_present(pmde) && pmd_devmap(pmde))) { From patchwork Fri Jun 9 01:08:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273062 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F38EC7EE25 for ; Fri, 9 Jun 2023 01:08:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1A9918E0010; Thu, 8 Jun 2023 21:08:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 132EE8E0001; Thu, 8 Jun 2023 21:08:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EEFB08E0010; Thu, 8 Jun 2023 21:08:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id DD1C48E0001 for ; Thu, 8 Jun 2023 21:08:27 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id AFE47AEEEB for ; Fri, 9 Jun 2023 01:08:27 +0000 (UTC) X-FDA: 80881423854.04.D6D0A6F Received: from mail-qk1-f179.google.com (mail-qk1-f179.google.com [209.85.222.179]) by imf23.hostedemail.com (Postfix) with ESMTP id E67D214000C for ; Fri, 9 Jun 2023 01:08:25 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=4TUvz+8K; spf=pass (imf23.hostedemail.com: domain of hughd@google.com designates 209.85.222.179 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686272905; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=howkEJoV//tXil1Z2yV8DA3fQgmxIGa28HiuE8oWzhc=; b=SmT7bk4ESD3pDT0VWHhlQm6CxSUCCJOSqFue4WKrNgXMo3AD0Wqh2GdFU1NzphCiYymOn/ Efht09nWqImgeibv4aG/HosxLxEGAGM0DcuJDIr11YaRBypjO91aRgZweh2mMQ+BHH3J/b WDCnY0gy64N2V1C/48R0h3U0H3MSGo0= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=4TUvz+8K; spf=pass (imf23.hostedemail.com: domain of hughd@google.com designates 209.85.222.179 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686272905; a=rsa-sha256; cv=none; b=mhiZuPtSDoaMIsWtK2XefdFQ3aPwgbnj+q89TaJuUpTOx0XoStOz6YRxcW4W2oo9x30fsC IaUY92F1+gSfANzNANaEE3aWdnNUr0Lw6kfPCVmKVKKhIKwvs/mAHcXDUnWbpwKj0vdzNI UZvJBVF5lUMCt678blTD3q8eKsIE8cU= Received: by mail-qk1-f179.google.com with SMTP id af79cd13be357-75ebd39fac8so105222885a.3 for ; Thu, 08 Jun 2023 18:08:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686272905; x=1688864905; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=howkEJoV//tXil1Z2yV8DA3fQgmxIGa28HiuE8oWzhc=; b=4TUvz+8KoUFPjBrCAcYR59EYGfTLuV6CLia8XZUURQDZbunBGHJYb+jGkZmsygXgl3 +15vELNZ2tt6xf6nRk+326C0YlS5yYQ2GCB6uL+UjODH+sl0FpGKeWODfJXUrOGUmWtC s13E9TqS54wkrJ0CBEm34+uvV622uJ8q5jR+3xcY9g99e07dWOHLQHDJc1AmZlHp7BF3 G5Ij1yJL5CY3uB6yvHN60sdMB/duTTXIRCBGPDCpPIbQNiuj+9xk+3ynZo+U+EWgi22V hQAEa6GzzVBUVKOsI+w/Ikakst0ZV10IGK7MsNaGUrB0ra9/vx96mtgIfR6mIWCjP/h4 L8aw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686272905; x=1688864905; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=howkEJoV//tXil1Z2yV8DA3fQgmxIGa28HiuE8oWzhc=; b=Z2z+CTmGHaViVGzuEyQQjq4AT5viLu0pJvfAjrz2h/Bi5F1qt4kp/Ag2vOrkVZz82m qSRS2NcoFaI8hgBj9feRlX7nTidMZSoAhC63y25dBySoKDE7TPM3OzjH+ugCaaBaL2BF aVH4uGGylEvZZnKMN+lLtewHMo1+vYE/bDaPJ9OkWOUcFNOOXn1sSH+oSOKVxx9vsIgM K8/lVDPq3BqdXHDzzMvJ7Vn1n3YQYNOPet0rR7ecyfI5V4RLnSvdUpV94YtXKcDuhKO6 XP5tPPbyGg4U+qbLqQshqvK7wXZKg/rLv+HSKNdnX/sxf3FdK6occMQqlBgGAsMUTOZX 1hTQ== X-Gm-Message-State: AC+VfDyTjILWWCCX4ickcFpuskj2iCVCTwPiZWSLJwis8SXLMLyFnhg0 hGf3jtgB2AuURAl3Cr6oTS1PWQ== X-Google-Smtp-Source: ACHHUZ7/PxpX5u49rxcuOgTsJSbOZnLq1smZ1N/fcsjFrRWYRcflJMGjskZfjY8eigXMifdaD6g1aw== X-Received: by 2002:a05:620a:84c6:b0:75d:5571:64c0 with SMTP id pq6-20020a05620a84c600b0075d557164c0mr6330539qkn.37.1686272904886; Thu, 08 Jun 2023 18:08:24 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id x204-20020a81a0d5000000b00565de196516sm286345ywg.32.2023.06.08.18.08.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:08:24 -0700 (PDT) Date: Thu, 8 Jun 2023 18:08:20 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 02/32] mm/migrate: remove cruft from migration_entry_wait()s In-Reply-To: Message-ID: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: E67D214000C X-Rspam-User: X-Stat-Signature: ja5ccreciefttn598scj46bj5faprk18 X-Rspamd-Server: rspam01 X-HE-Tag: 1686272905-570598 X-HE-Meta: U2FsdGVkX19UQ/NUnzBbboUpZMzERFVPmyB5qy40XT3NxDWU9GC9qOffFBzVSrVgK/ty4PkQYg2LiYE0LBjYjtKseAi0yQIV+D/Ury7tSm3dmYe1B/ACBaH55A7cX7eM1soSEXNK3QpwG2yydmvpmNfZD7JvYX9xFAe87qdwQnajpNEVsPzNTve2jk6LWcfrRKcvSV8UXOwQHQbis/sRIrhI6bLgVMGcApnM0klmm4jxEpZVcDQ8P8zFxo2hivSKxAdma0q6zda8XngR5meDFw2u5j6BxAwcKvSzx4P5Q+e8JZhgncTrVEzy/ntR6aLeW1Wj/5EW/MI4UaCwfzlgjPbhLZzhgdR+NpRpsgi/JVlIWY3jqBydqIz1y2jnQ/GEvedcSmHNpniIRD16hTvFnp3r5Ay3GWe9fXUHf9Ce4lHK6qgxZaKZvxiGDQOsJKOCgRJG7PrfIDsSkKUmhk2vaQurBDQgMWIPBK9D+dNXAnIUuDwymuAm7EyKm9unBEVW2wTtjggAMCoALvIYjjp8Y6jHge/+nek3zwPcZInO2F/3aOkdAJ4e7GXaUmQ0cO2gIKK9W67V8sMDo6vHhQcRsnkP9KTGV4IU3iiOO8ZYePuy0vzettjQUzJ8TXQrsSchVmFxQM63dslcp/sM8wPYVnBiRv0cE4rlsPwy/yPZK9FDQ3CAR6iN34jlAsPdmbnhauvA/XNKwECyz0UDK0NGoNs4SRuApKrbHG0EyqlLL1h2X+gunoSCvEqj+GBzBnkbP7wZhvgHeYzuIMz3Wq/+211bQiegCaOl6IKFPOIkdUtrqdNatzx9WJKb2kCuZYolk0a2Ii1z1keDDLwye/tqK3PolXWwolKj8t9JwtDHefYfAbl0QnS29YgRPrmtPrP4DaPNKUfg+RyyjEbjuu3d8INm/gDHxVpONqpWW/1SYqVzPDkOaEkiuy4qH8+R8NLiGFDbxZ3aUR3Sg44Pleg J8F5F9x+ fRT7QFr0tCGnhke5pwUpUCPAw8u31jLyc7vfqSEikzR8eQdgOZoVf8IuK+zlTW02WkR/rNGJit7BsUDd5XGT8l0OiQQR+8aGHbQYglj3g4MMee3KRaNP2Om6vjUY/AMepNltLHyE9XBmZWrr1yjNg8MlQ9T8w7JzD1gGk3NjGFxuBeq5h/cdwvZTwcEe50vSwbKMi+IeOobE9L5s8McpVlBNyLvU0mV7KpoUHmjP48Qq8n45g6V8bRbwF050o2MRcfw5rFOFTWt6uuKq546IUve9jUHGA5CNzoCx9HFaUWH+Pqqb1NSbZz/uUTpHQSmLJ/KkmTdC4ZjocWYHu8KZRxpyKvNQuSKQJ0dEfUrxFkBozZQJPucXz95UUZDd8LWOdxHEWwHTgd0LQfcqJ71Q5hpONz5kFUrmUOZ8dpwUoK7cWq1xkDxBudeKbdu/8cy3xDG4B4P9i4bRWc+A= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: migration_entry_wait_on_locked() does not need to take a mapped pte pointer, its callers can do the unmap first. Annotate it with __releases(ptl) to reduce sparse warnings. Fold __migration_entry_wait_huge() into migration_entry_wait_huge(). Fold __migration_entry_wait() into migration_entry_wait(), preferring the tighter pte_offset_map_lock() to pte_offset_map() and pte_lockptr(). Signed-off-by: Hugh Dickins Reviewed-by: Alistair Popple --- include/linux/migrate.h | 4 ++-- include/linux/swapops.h | 17 +++-------------- mm/filemap.c | 13 ++++--------- mm/migrate.c | 37 +++++++++++++------------------------ 4 files changed, 22 insertions(+), 49 deletions(-) diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 6241a1596a75..affea3063473 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -75,8 +75,8 @@ bool isolate_movable_page(struct page *page, isolate_mode_t mode); int migrate_huge_page_move_mapping(struct address_space *mapping, struct folio *dst, struct folio *src); -void migration_entry_wait_on_locked(swp_entry_t entry, pte_t *ptep, - spinlock_t *ptl); +void migration_entry_wait_on_locked(swp_entry_t entry, spinlock_t *ptl) + __releases(ptl); void folio_migrate_flags(struct folio *newfolio, struct folio *folio); void folio_migrate_copy(struct folio *newfolio, struct folio *folio); int folio_migrate_mapping(struct address_space *mapping, diff --git a/include/linux/swapops.h b/include/linux/swapops.h index 3a451b7afcb3..4c932cb45e0b 100644 --- a/include/linux/swapops.h +++ b/include/linux/swapops.h @@ -332,15 +332,9 @@ static inline bool is_migration_entry_dirty(swp_entry_t entry) return false; } -extern void __migration_entry_wait(struct mm_struct *mm, pte_t *ptep, - spinlock_t *ptl); extern void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, unsigned long address); -#ifdef CONFIG_HUGETLB_PAGE -extern void __migration_entry_wait_huge(struct vm_area_struct *vma, - pte_t *ptep, spinlock_t *ptl); extern void migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *pte); -#endif /* CONFIG_HUGETLB_PAGE */ #else /* CONFIG_MIGRATION */ static inline swp_entry_t make_readable_migration_entry(pgoff_t offset) { @@ -362,15 +356,10 @@ static inline int is_migration_entry(swp_entry_t swp) return 0; } -static inline void __migration_entry_wait(struct mm_struct *mm, pte_t *ptep, - spinlock_t *ptl) { } static inline void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, - unsigned long address) { } -#ifdef CONFIG_HUGETLB_PAGE -static inline void __migration_entry_wait_huge(struct vm_area_struct *vma, - pte_t *ptep, spinlock_t *ptl) { } -static inline void migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *pte) { } -#endif /* CONFIG_HUGETLB_PAGE */ + unsigned long address) { } +static inline void migration_entry_wait_huge(struct vm_area_struct *vma, + pte_t *pte) { } static inline int is_writable_migration_entry(swp_entry_t entry) { return 0; diff --git a/mm/filemap.c b/mm/filemap.c index b4c9bd368b7e..28b42ee848a4 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1359,8 +1359,6 @@ static inline int folio_wait_bit_common(struct folio *folio, int bit_nr, /** * migration_entry_wait_on_locked - Wait for a migration entry to be removed * @entry: migration swap entry. - * @ptep: mapped pte pointer. Will return with the ptep unmapped. Only required - * for pte entries, pass NULL for pmd entries. * @ptl: already locked ptl. This function will drop the lock. * * Wait for a migration entry referencing the given page to be removed. This is @@ -1369,13 +1367,13 @@ static inline int folio_wait_bit_common(struct folio *folio, int bit_nr, * should be called while holding the ptl for the migration entry referencing * the page. * - * Returns after unmapping and unlocking the pte/ptl with pte_unmap_unlock(). + * Returns after unlocking the ptl. * * This follows the same logic as folio_wait_bit_common() so see the comments * there. */ -void migration_entry_wait_on_locked(swp_entry_t entry, pte_t *ptep, - spinlock_t *ptl) +void migration_entry_wait_on_locked(swp_entry_t entry, spinlock_t *ptl) + __releases(ptl) { struct wait_page_queue wait_page; wait_queue_entry_t *wait = &wait_page.wait; @@ -1409,10 +1407,7 @@ void migration_entry_wait_on_locked(swp_entry_t entry, pte_t *ptep, * a valid reference to the page, and it must take the ptl to remove the * migration entry. So the page is valid until the ptl is dropped. */ - if (ptep) - pte_unmap_unlock(ptep, ptl); - else - spin_unlock(ptl); + spin_unlock(ptl); for (;;) { unsigned int flags; diff --git a/mm/migrate.c b/mm/migrate.c index 01cac26a3127..3ecb7a40075f 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -296,14 +296,18 @@ void remove_migration_ptes(struct folio *src, struct folio *dst, bool locked) * get to the page and wait until migration is finished. * When we return from this function the fault will be retried. */ -void __migration_entry_wait(struct mm_struct *mm, pte_t *ptep, - spinlock_t *ptl) +void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, + unsigned long address) { + spinlock_t *ptl; + pte_t *ptep; pte_t pte; swp_entry_t entry; - spin_lock(ptl); + ptep = pte_offset_map_lock(mm, pmd, address, &ptl); pte = *ptep; + pte_unmap(ptep); + if (!is_swap_pte(pte)) goto out; @@ -311,18 +315,10 @@ void __migration_entry_wait(struct mm_struct *mm, pte_t *ptep, if (!is_migration_entry(entry)) goto out; - migration_entry_wait_on_locked(entry, ptep, ptl); + migration_entry_wait_on_locked(entry, ptl); return; out: - pte_unmap_unlock(ptep, ptl); -} - -void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, - unsigned long address) -{ - spinlock_t *ptl = pte_lockptr(mm, pmd); - pte_t *ptep = pte_offset_map(pmd, address); - __migration_entry_wait(mm, ptep, ptl); + spin_unlock(ptl); } #ifdef CONFIG_HUGETLB_PAGE @@ -332,9 +328,9 @@ void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, * * This function will release the vma lock before returning. */ -void __migration_entry_wait_huge(struct vm_area_struct *vma, - pte_t *ptep, spinlock_t *ptl) +void migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *ptep) { + spinlock_t *ptl = huge_pte_lockptr(hstate_vma(vma), vma->vm_mm, ptep); pte_t pte; hugetlb_vma_assert_locked(vma); @@ -352,16 +348,9 @@ void __migration_entry_wait_huge(struct vm_area_struct *vma, * lock release in migration_entry_wait_on_locked(). */ hugetlb_vma_unlock_read(vma); - migration_entry_wait_on_locked(pte_to_swp_entry(pte), NULL, ptl); + migration_entry_wait_on_locked(pte_to_swp_entry(pte), ptl); } } - -void migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *pte) -{ - spinlock_t *ptl = huge_pte_lockptr(hstate_vma(vma), vma->vm_mm, pte); - - __migration_entry_wait_huge(vma, pte, ptl); -} #endif #ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION @@ -372,7 +361,7 @@ void pmd_migration_entry_wait(struct mm_struct *mm, pmd_t *pmd) ptl = pmd_lock(mm, pmd); if (!is_pmd_migration_entry(*pmd)) goto unlock; - migration_entry_wait_on_locked(pmd_to_swp_entry(*pmd), NULL, ptl); + migration_entry_wait_on_locked(pmd_to_swp_entry(*pmd), ptl); return; unlock: spin_unlock(ptl); From patchwork Fri Jun 9 01:09:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273063 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10BD1C7EE25 for ; Fri, 9 Jun 2023 01:09:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8FBA38E0011; Thu, 8 Jun 2023 21:09:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 885338E0001; Thu, 8 Jun 2023 21:09:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 74CCA8E0011; Thu, 8 Jun 2023 21:09:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 66DE78E0001 for ; Thu, 8 Jun 2023 21:09:33 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 399B5C0311 for ; Fri, 9 Jun 2023 01:09:33 +0000 (UTC) X-FDA: 80881426626.08.A2CE40A Received: from mail-vs1-f54.google.com (mail-vs1-f54.google.com [209.85.217.54]) by imf25.hostedemail.com (Postfix) with ESMTP id 61959A0005 for ; Fri, 9 Jun 2023 01:09:31 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=UcfhhEB8; spf=pass (imf25.hostedemail.com: domain of hughd@google.com designates 209.85.217.54 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686272971; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lc/g6LXQOs93yTgCiGczTkkFK+sAgodNjTysgahDEVw=; b=ZkzgiNfA0fUKn/lPa8uWgVy+RDPp6bPu1uruMT1p6PTjZXRY0KIibP+uelIgroK8gZQNTq Rnw7LfhUzbDRjO0JsB/32t//AFiimQEw0fPOpKC4jdPvNYk/Bsvep773TCBR0MwIrG3g1F ugV4RW35B8f0V4k/VUJSy3tEBSKpyeY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686272971; a=rsa-sha256; cv=none; b=4HWFBw6tvoa4zcf4w1g7PIwSTUv8L16TTllDwBKXAMz0LXKcTxldN3rcO8m3Tdo958sC0z j9xP0w1E1r/vqRtiLXoSfjLmP+M+MNN8kCz2a7CAS7WseTWybaElPuDma/MSndIlaOMgau ttVfYnYdf0HGgk8icQGp86VkfYZy5wA= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=UcfhhEB8; spf=pass (imf25.hostedemail.com: domain of hughd@google.com designates 209.85.217.54 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-vs1-f54.google.com with SMTP id ada2fe7eead31-43dc7a78a37so401827137.0 for ; Thu, 08 Jun 2023 18:09:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686272970; x=1688864970; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=lc/g6LXQOs93yTgCiGczTkkFK+sAgodNjTysgahDEVw=; b=UcfhhEB8JVYtSaYNjeXN56wB9MqyMQtCQGtprQFTqjfYjEL7E7V3Iy94OMfqel1oXO QZIcQiOjvsa3e98AoL/nIYiIeV2XwYtBY8pjjhbDjSLvNRXQUNuQ1CEyYWBCd3xYPZVP trZtykP/Mof04N1VHc0lVHbgyp1EiW4pew5nO8aSYUb++iO0ncGeIIIiQK3p/c+W0yHU AAa6tsu0E2vKcj4HqLxwkHvPM9meFiA6K3qv+5vTGLBRG4cfgXg5/75cLxanqJajUo8o cFN/FNCedO3dWchgzF6syrkwFnGXauAim52zAU4uX164Btb5VfynbXsQAJloguXMP3eG dROw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686272970; x=1688864970; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=lc/g6LXQOs93yTgCiGczTkkFK+sAgodNjTysgahDEVw=; b=ELiir/axakDBVlnjq0w5hcLobWnIJc/YyCjHkpyDadn3bhK3CvnqXlf25pco+UPB8b tYDmzGBkG8wzm5IP7rcq7zgTPCeKhV4X9vrQ1IqxULQ0pd+LtBPNo9Qgf9npAei1IyBw izOozbuJXZRxm+a4g+f5UXsb14JEMwdVOc4QqSyxYw99oeSMf/BSlGynHWXiNBxgSaNu qCwpoln8lTih6+Vd+3xE/NekDSXynHvbmg2SKLo/GW5M3yaouxgDSVrJ5nDwm82Ib0wm jz8W8nxkocw42IohP/O16O8MJZXTgA2TONocUYZ2zvTktSLASHTG3r7Bz1x+rFF90ywF qGEg== X-Gm-Message-State: AC+VfDwTYV2WYk5/3B0M2VNdKz8Bh9vbaAdr6vqohWMfeYcWtk6N9Mji CpoJLGhx8Oi5lP1PIkOTbtN7Pw== X-Google-Smtp-Source: ACHHUZ5C3ZXPkFDl7beNnC8Wxo2aSMU5SCcCj/7iCu4IfWqU05s5B/rb2VYjxSOJxyLyi/equlOn5A== X-Received: by 2002:a67:f490:0:b0:43b:1dfa:2534 with SMTP id o16-20020a67f490000000b0043b1dfa2534mr162704vsn.10.1686272970295; Thu, 08 Jun 2023 18:09:30 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id n11-20020a25400b000000b00bb393903508sm603507yba.14.2023.06.08.18.09.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:09:29 -0700 (PDT) Date: Thu, 8 Jun 2023 18:09:25 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 03/32] mm/pgtable: kmap_local_page() instead of kmap_atomic() In-Reply-To: Message-ID: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 61959A0005 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 8c5hmat6nxai9oxzs1zws1by1soi9p5a X-HE-Tag: 1686272971-562033 X-HE-Meta: U2FsdGVkX1+yEgVg3XwGUBl9S+1i2Ry5mPmJviSheANUclbxt6tLPDbAZB6YvbBOR+sbBWOnLkxzDhYL/1Ufm3YW5YIH6uqP6nV5MpWz/i0uM1dQulyiRgkx5Lf7VX0CrSmy6ATEVFPTAXcTaFHGgYUFtngLYUi37wUWMVrj8dksTu/dP39969ymYlM8luuV/5wMr5bciQneuemhwnP0//H8mPV5qJh7nImOz/B3La7kAcl+gHi6yKU7lfpzAgRRvvsdpABOhCsWM0negzIRWXikMs8hK7U6z88VXdLbqhf5dPdyanlGudL3Uhp8f+uAKSTRf0C9J+/L8nW3K++u2U0RATBPfE8q9DZcKz4WKlxQR5pUTUxxQm2uUd/W0eIdkS/IDbaOCAS5KxpZvFjoYf///OiGCor32ubUBvN5VJGCQehI/mCYpeUrGY/jWu6SU/1CC8w4KHCWJCupRGeFWDzrOC38zQDPVx0d22BEbfqaXktGpq+M6m+wFvCpCGliWtuGjBvKtMbFhqe5Qta6xEzLrzFf1R2sWDofy6BQ36RpvCWhCGq/ppAXFwLdRh0eaki9qNuPhQoZXzVsngeQD/Cbtb2nA28vdcgM4XlK5/otsI968BQU0+DvulT74DceDGqxS6vV2OyHnMsHktfTt4lhn1iULTpq7RiyaPC+GpAKTUraItt2SHC3/ZvuuzdoDp/x+B/8qdLgdBkXm9kLn9YiPPAVPtT+EenIzeDZuCesZsdalq20YgqB6K+xecnNLxU2uWB6RwdiwQdmkol6ATbmfRH7nMRnKjMuHHZohDxhf0ukLgAF08WINRrRd4Vtt6Df4fA6W179FejLdlAjrgWBts7tfJm7SrXFNskHhDGq83UYmEWffjgjrIyZgsGVamP4NY3RoRENkJf05gjBTlYN771lsPgljpA92PD0q3OM0I3R/bU5CEnZGqy8Cn/ZwmqHURE6rRu0+TS8CM4 2J3KQWBx YAFvCArn4C0KqClvCXNbBXpJyDxne1qLCXIpn7UnosDo2FnpG5G7ORVWMziUS4jGTCAqdWJpUy9ydE/ZgXtJrQzgr8vd3GUs4XIOxJxaeMQgsN6uiAIj/DAHfm1HyJo/GNvgXnkm/zssecxeRLcnEBK3h9LS2d+nPI24GQlklFkWKnxsSHFjrtCeEJEECc9qpEF93MgyAD3w7VAqK4AduMNzIzmxLfw0tL+g7Hhqu9Kxq6wKYD8rgIteYdz8rQjpjSWMC4+S6D5wS5LO5J5mu4IfyNUMXQ3NxmR7sOgRqaQneTqVVbcnYbI7IdhZt0YIQ9tN+fi/7OHXLmaTkz3ciIoL1ceJeRpcMbgzowGemDLLJJg/E6CAxCmaU4/PwyX+Flh4foUKoDUQwsYOtwHAnX9ARLYA72oUPJ4C4fQP/1isNUMRVkPMT1lxJtg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: pte_offset_map() was still using kmap_atomic(): update it to the preferred kmap_local_page() before making further changes there, in case we need this as a bisection point; but I doubt it can cause any trouble. Signed-off-by: Hugh Dickins --- include/linux/pgtable.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 8ec27fe69dc8..94235ff2706e 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -96,9 +96,9 @@ static inline pte_t *pte_offset_kernel(pmd_t *pmd, unsigned long address) #if defined(CONFIG_HIGHPTE) #define pte_offset_map(dir, address) \ - ((pte_t *)kmap_atomic(pmd_page(*(dir))) + \ + ((pte_t *)kmap_local_page(pmd_page(*(dir))) + \ pte_index((address))) -#define pte_unmap(pte) kunmap_atomic((pte)) +#define pte_unmap(pte) kunmap_local((pte)) #else #define pte_offset_map(dir, address) pte_offset_kernel((dir), (address)) #define pte_unmap(pte) ((void)(pte)) /* NOP */ From patchwork Fri Jun 9 01:10:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273064 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77BF9C7EE29 for ; Fri, 9 Jun 2023 01:10:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1819B8E0012; Thu, 8 Jun 2023 21:10:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 10AE38E0001; Thu, 8 Jun 2023 21:10:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EEDC38E0012; Thu, 8 Jun 2023 21:10:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id DFE428E0001 for ; Thu, 8 Jun 2023 21:10:39 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id B7E25A024C for ; Fri, 9 Jun 2023 01:10:39 +0000 (UTC) X-FDA: 80881429398.13.23EFF28 Received: from mail-yw1-f175.google.com (mail-yw1-f175.google.com [209.85.128.175]) by imf27.hostedemail.com (Postfix) with ESMTP id D4ACD40010 for ; Fri, 9 Jun 2023 01:10:37 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=pehkqmIu; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf27.hostedemail.com: domain of hughd@google.com designates 209.85.128.175 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686273037; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=aQBwLL4HlECbSO5gR059+G6dephAddbWEgO6E2LDWZA=; b=yS8pu6+XiYGKeyOS5ty3mHqdDox/DF2583G1JNdr8OK4mLZ1wLXds/9dkshRtMPQEQZs0V XUb8PJUwdEfFlOyzwb3fp4zRSCLAFTXywUZtKi+CjM/0yxdO0V8mGCAjGNDbq5ONmDD7t0 0pV9ov0arMzcawqmOdyxTuyTBw/Y01Q= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=pehkqmIu; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf27.hostedemail.com: domain of hughd@google.com designates 209.85.128.175 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686273037; a=rsa-sha256; cv=none; b=baXfxd6NayDjtDc2Bac/BIHPt8bu2dE36JiQ/XvQ2AI3TXirWZ8RTn7bLa7sLDYz7Ey+SO kqyrgMMJboNdDWsBSHpz0bDEymzSn3PNoqf0PbtW05HZTCXWe+oeiS6V2IDomZl3Jzkwb9 cxSbcfm7dr262H0dqbtbtyYzcdmnSl8= Received: by mail-yw1-f175.google.com with SMTP id 00721157ae682-5659d85876dso10691597b3.2 for ; Thu, 08 Jun 2023 18:10:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686273037; x=1688865037; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=aQBwLL4HlECbSO5gR059+G6dephAddbWEgO6E2LDWZA=; b=pehkqmIu5uuC2/B3POt5qXky+VS7UB4H0MKZKoA0UPHDqWhVIxnSWvN2qZj214i7Wi xlP2w3eBKNw5TWHpEBt8Ykvx85JhHDfFCoQrHuZvQV3gF+6AdXASRaQWOncyvod5v/cn lgCa62bmqNVztpuFb1tB0iswKKgr36t6UpOuDeHZ2gPb/O3B3TO2wbqCc1yTI1noFll7 3JosUpTyMIooa8ybR3xoSKQBoApqEb5yRt0CGd4bkYJ/922RqEv0JAPqXk9hzJKhIMnT DFobijIf2lK30P2q6urus71da4ADA/iomZybOe90CAfsWC55Rzl6LoyMj3bmbkCn6ZM3 5Fuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686273037; x=1688865037; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=aQBwLL4HlECbSO5gR059+G6dephAddbWEgO6E2LDWZA=; b=Fyez0RhrbYY3ZLUJ8Y1oosIi1e2RhyvLLiX3SLMHphWM1bNWGh/wbRmXXQtnJusocw tKrKDwfH/oaCBeCHnVavtiLCsvc/DFnJMhNgaD8CaskC/O0It7t4SZchfow2dc9jPK8G W9XFXH7Tx2Fv+Lm7LbAvZ53nW2hK+fQ5WNy8bYmAceEer3m5zST906iHW2EDTMrBHfpV qFaOox5C4BHWTAJat+S4A+MGkFruNhrWbqPXgmOqazDK+EB8ECzhztzww1pQQEoEhHJm umM4KpBHKbwZES0YbyK6XQJZJsY6lAM9iMJ75NqCyPe1ueGPAtjZx2W9BO+UgrW3tRsN Itvg== X-Gm-Message-State: AC+VfDxwUwCxUCwrkX8KbizztlBbgWQrJZcYzUIBMhAcDFVZAXUcou1m 5xBS+7Zh6NGAimwFOUklKaM9Sw== X-Google-Smtp-Source: ACHHUZ4RPJYEqOW+mRJ0KmB3JL8CSbjs+Mlo2tkty+xU7pqquzaJQl3sBeIwjaFrTRmmS8IvpanSCg== X-Received: by 2002:a81:47d4:0:b0:565:9d00:8a14 with SMTP id u203-20020a8147d4000000b005659d008a14mr1378760ywa.20.1686273036762; Thu, 08 Jun 2023 18:10:36 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id a17-20020a81bb51000000b00545a08184fdsm265795ywl.141.2023.06.08.18.10.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:10:36 -0700 (PDT) Date: Thu, 8 Jun 2023 18:10:32 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 04/32] mm/pgtable: allow pte_offset_map[_lock]() to fail In-Reply-To: Message-ID: <2929bfd-9893-a374-e463-4c3127ff9b9d@google.com> References: MIME-Version: 1.0 X-Rspamd-Queue-Id: D4ACD40010 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: snu8rie3uf56u13dyb3fii9fgcrxsmwd X-HE-Tag: 1686273037-656831 X-HE-Meta: U2FsdGVkX19xFOQALqFKqbs7YSwmOuZu7wymezE3qenEUL7gYlxHdULPg6Z3r04mXbk4rtJAFqAiZSa48qEMWLgDmFG3BStZThw40pJ0PlnhGHsz0Zk1M1t9Xd9cfk36NCcWFS9cYJf25Y+UXglETEqw/vo80J5+nxavRFhAcMT4M2nunskXh08uIiIw1dtN1bBQYugs9uTbb3DCgyAv1ZAY/tmp2nb26VBjMMnZ0p+ArpWy3tNETVS7Bt8PzFE2iwYNdTRBZs1P/OFrg+pdumYdgpxhmk/dZPRvH+LImXp84kKcIDYZvDm0iUG62gsilvjTeqPpH9jmCMdVO6vYLAewoaWA2QOudWog1ZFohzfOzylWA/Zv0ccRhy8nV4Liy1c2LUzPTwJfsitPgcm+WJ4Rdqy3lYlxLbGEGn0x6ixNKUA+oiK9TSgH46sUO3DobzkMg0gQPSWfXEOKkraOWwUA2YWzkmNi2MobvpXsZQnBM6BulpKJrfNsXfgLp5EubbY0emvfkfb3fjr1nSSTZu5PbKAsEoiz5CyOfpzSpGQNDxgXZkKYYXX0sXqppvQDzAqddFbAvytOzjj/nnDftwtexrTT2YikZDfNseJMQ2frzHpkjzlO/cqiJ/pqt7YPMlQU9BLBN/OS1Olo0b4HlYF62uFObwLWTpy2pQnvr/aH62f0DH1of3aOjdnt1YnZI5e6c9hmOk3QY0RuhexsZdNX9XxB0T3TfKVWzTqFs58XNUXAZKNWWYC22ZOtbOTbdR3Nmlv2m6Hc0/bLKrdaWa8GKSYEltbob/r30FUWqeMB54GRPfr62dsTDqI+nw0MjukF08OpBVX406+R86NeD52PY68anDJlbmjXl0d01FOEpqPy5uFLZYmjQo8Nfr3dkLHhge8OmQCGS5tORMnzXGqRgHlC+oVvd97MifX4FYmBOCVAsECgQLDupAYoWTRbyYi08tTJG/ukSHzeyjV Yk/WUGoy r1oZ6gUuPa4Q9U0ulIpt5EsnfEEXuVgqByJEp40V+fLfMqqJX+thCrVSxisLaGOwwMGv7LW/ypRjHtM0DRWFnS2qkolkPRq/Wv4gw+HJc4JX1dQiBZ7VRoMx0i9w6fQHq9DvCkqYxn5FmgnU0W1ewoSNESbuK+8wj87OG6Rzyh9hPJEOIYHvnABmJ6oQPCs5csVNsFziidvcKIYwfhkCUj3CDFj7EndZzbBTHfxxLZRvewudlq/6gwXtTKw3zdGVhdCUD9wH5gyQ9vDB5y/onxWm2iKEyj+ac1rZ/0HrK+kqoPlbJE8/UHpAxkWezF1P8xkslq7GAwujbcSymBZAFx7NneDi3CZzeEJIKt0XBFZifFSS4oE7LkRuNjuB2nQw/ShCPZYWehAdzACYpGnFWmT6gl+cmsnYUfOIKscqw0sXPDcMzWnFxEHl2Mw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make pte_offset_map() a wrapper for __pte_offset_map() (optionally outputs pmdval), pte_offset_map_lock() a sparse __cond_lock wrapper for __pte_offset_map_lock(): those __funcs added in mm/pgtable-generic.c. __pte_offset_map() do pmdval validation (including pmd_clear_bad() when pmd_bad()), returning NULL if pmdval is not for a page table. __pte_offset_map_lock() verify pmdval unchanged after getting the lock, trying again if it changed. No #ifdef CONFIG_TRANSPARENT_HUGEPAGE around them: that could be done to cover the imminent case, but we expect to generalize it later, and it makes a mess of where to do the pmd_bad() clearing. Add pte_offset_map_nolock(): outputs ptl like pte_offset_map_lock(), without actually taking the lock. This will be preferred to open uses of pte_lockptr(), because (when split ptlock is in page table's struct page) it points to the right lock for the returned pte pointer, even if *pmd gets changed racily afterwards. Update corresponding Documentation. Do not add the anticipated rcu_read_lock() and rcu_read_unlock()s yet: they have to wait until all architectures are balancing pte_offset_map()s with pte_unmap()s (as in the arch series posted earlier). But comment where they will go, so that it's easy to add them for experiments. And only when those are in place can transient racy failure cases be enabled. Add more safety for the PAE mismatched pmd_low pmd_high case at that time. Signed-off-by: Hugh Dickins --- Documentation/mm/split_page_table_lock.rst | 17 ++++--- include/linux/mm.h | 27 +++++++---- include/linux/pgtable.h | 22 ++++++--- mm/pgtable-generic.c | 56 ++++++++++++++++++++++ 4 files changed, 101 insertions(+), 21 deletions(-) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/split_page_table_lock.rst index 50ee0dfc95be..a834fad9de12 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -14,15 +14,20 @@ tables. Access to higher level tables protected by mm->page_table_lock. There are helpers to lock/unlock a table and other accessor functions: - pte_offset_map_lock() - maps pte and takes PTE table lock, returns pointer to the taken - lock; + maps PTE and takes PTE table lock, returns pointer to PTE with + pointer to its PTE table lock, or returns NULL if no PTE table; + - pte_offset_map_nolock() + maps PTE, returns pointer to PTE with pointer to its PTE table + lock (not taken), or returns NULL if no PTE table; + - pte_offset_map() + maps PTE, returns pointer to PTE, or returns NULL if no PTE table; + - pte_unmap() + unmaps PTE table; - pte_unmap_unlock() unlocks and unmaps PTE table; - pte_alloc_map_lock() - allocates PTE table if needed and take the lock, returns pointer - to taken lock or NULL if allocation failed; - - pte_lockptr() - returns pointer to PTE table lock; + allocates PTE table if needed and takes its lock, returns pointer to + PTE with pointer to its lock, or returns NULL if allocation failed; - pmd_lock() takes PMD table lock, returns pointer to taken lock; - pmd_lockptr() diff --git a/include/linux/mm.h b/include/linux/mm.h index 27ce77080c79..3c2e56980853 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2787,14 +2787,25 @@ static inline void pgtable_pte_page_dtor(struct page *page) dec_lruvec_page_state(page, NR_PAGETABLE); } -#define pte_offset_map_lock(mm, pmd, address, ptlp) \ -({ \ - spinlock_t *__ptl = pte_lockptr(mm, pmd); \ - pte_t *__pte = pte_offset_map(pmd, address); \ - *(ptlp) = __ptl; \ - spin_lock(__ptl); \ - __pte; \ -}) +pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp); +static inline pte_t *pte_offset_map(pmd_t *pmd, unsigned long addr) +{ + return __pte_offset_map(pmd, addr, NULL); +} + +pte_t *__pte_offset_map_lock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp); +static inline pte_t *pte_offset_map_lock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp) +{ + pte_t *pte; + + __cond_lock(*ptlp, pte = __pte_offset_map_lock(mm, pmd, addr, ptlp)); + return pte; +} + +pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp); #define pte_unmap_unlock(pte, ptl) do { \ spin_unlock(ptl); \ diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 94235ff2706e..3fabbb018557 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -94,14 +94,22 @@ static inline pte_t *pte_offset_kernel(pmd_t *pmd, unsigned long address) #define pte_offset_kernel pte_offset_kernel #endif -#if defined(CONFIG_HIGHPTE) -#define pte_offset_map(dir, address) \ - ((pte_t *)kmap_local_page(pmd_page(*(dir))) + \ - pte_index((address))) -#define pte_unmap(pte) kunmap_local((pte)) +#ifdef CONFIG_HIGHPTE +#define __pte_map(pmd, address) \ + ((pte_t *)kmap_local_page(pmd_page(*(pmd))) + pte_index((address))) +#define pte_unmap(pte) do { \ + kunmap_local((pte)); \ + /* rcu_read_unlock() to be added later */ \ +} while (0) #else -#define pte_offset_map(dir, address) pte_offset_kernel((dir), (address)) -#define pte_unmap(pte) ((void)(pte)) /* NOP */ +static inline pte_t *__pte_map(pmd_t *pmd, unsigned long address) +{ + return pte_offset_kernel(pmd, address); +} +static inline void pte_unmap(pte_t *pte) +{ + /* rcu_read_unlock() to be added later */ +} #endif /* Find an entry in the second-level page table.. */ diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index d2fc52bffafc..c7ab18a5fb77 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -10,6 +10,8 @@ #include #include #include +#include +#include #include #include @@ -229,3 +231,57 @@ pmd_t pmdp_collapse_flush(struct vm_area_struct *vma, unsigned long address, } #endif #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ + +pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp) +{ + pmd_t pmdval; + + /* rcu_read_lock() to be added later */ + pmdval = pmdp_get_lockless(pmd); + if (pmdvalp) + *pmdvalp = pmdval; + if (unlikely(pmd_none(pmdval) || is_pmd_migration_entry(pmdval))) + goto nomap; + if (unlikely(pmd_trans_huge(pmdval) || pmd_devmap(pmdval))) + goto nomap; + if (unlikely(pmd_bad(pmdval))) { + pmd_clear_bad(pmd); + goto nomap; + } + return __pte_map(&pmdval, addr); +nomap: + /* rcu_read_unlock() to be added later */ + return NULL; +} + +pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp) +{ + pmd_t pmdval; + pte_t *pte; + + pte = __pte_offset_map(pmd, addr, &pmdval); + if (likely(pte)) + *ptlp = pte_lockptr(mm, &pmdval); + return pte; +} + +pte_t *__pte_offset_map_lock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp) +{ + spinlock_t *ptl; + pmd_t pmdval; + pte_t *pte; +again: + pte = __pte_offset_map(pmd, addr, &pmdval); + if (unlikely(!pte)) + return pte; + ptl = pte_lockptr(mm, &pmdval); + spin_lock(ptl); + if (likely(pmd_same(pmdval, pmdp_get_lockless(pmd)))) { + *ptlp = ptl; + return pte; + } + pte_unmap_unlock(pte, ptl); + goto again; +} From patchwork Fri Jun 9 01:11:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273066 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D4CCC7EE25 for ; Fri, 9 Jun 2023 01:11:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2B9CF8E0012; Thu, 8 Jun 2023 21:11:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 269F68E0001; Thu, 8 Jun 2023 21:11:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1323A8E0012; Thu, 8 Jun 2023 21:11:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 0621A8E0001 for ; Thu, 8 Jun 2023 21:11:37 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id DBD0F16026D for ; Fri, 9 Jun 2023 01:11:36 +0000 (UTC) X-FDA: 80881431792.29.279DC89 Received: from mail-qk1-f178.google.com (mail-qk1-f178.google.com [209.85.222.178]) by imf11.hostedemail.com (Postfix) with ESMTP id 198A84000C for ; Fri, 9 Jun 2023 01:11:34 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=sgrtTue3; spf=pass (imf11.hostedemail.com: domain of hughd@google.com designates 209.85.222.178 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686273095; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DKO2MGlysYL+7+oQy8UdaapVjC7HViDdumwjMefUUFk=; b=KUKwx9dBOJiqmYPGK1C8IK5r+dVWsnE4qYtLcc7IX3Gu5+Zx1TkqlxlNXSRyeFaouWGhIx AlQmpTUCOE1beRfliXi2rXvz8EpqudA5gEHGfBWf1kKsph7EeEb6Sj6jZgTpV70gYJLKiY cADvjje82+LYpHMiqpjPwJ0WiukrfEM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686273095; a=rsa-sha256; cv=none; b=h6k6atARq5+MUNyEhzlkPHt0Y1CYbITIwze4CBPWLpNQjSyBtl5hDcXeuEP9bkjvA+FXGm OlkTEX3EMsdEE0tj3aRGX7BzaRfMEKku/NZYRvnuBDIK62fL2APZYmASevFDqntLMpUYxl uqC2FqdJYI13njHtR4GrXAYbt7VSMBI= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=sgrtTue3; spf=pass (imf11.hostedemail.com: domain of hughd@google.com designates 209.85.222.178 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-qk1-f178.google.com with SMTP id af79cd13be357-75d3f2c9d13so118226985a.1 for ; Thu, 08 Jun 2023 18:11:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686273094; x=1688865094; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=DKO2MGlysYL+7+oQy8UdaapVjC7HViDdumwjMefUUFk=; b=sgrtTue3rv24LYzFS1k289BQNEsOuAchrD6cHOusSyoPb1K73NZICBU4ccdS+MDThN zBEewtmtI4CNAWNB55+HxFZZrtyp16opIUtn0ashIh7fwhjdgk+9O/mwLbz9OSd84hi/ pBLFfsBUzEB4I2fW6b9r72oAK2m+QnFCxV5NwJBtw2NTKg7epNTeTUcPNH+shND90LBV x2i2pLMwvamhFrNXJIWV1Y51CIqpk9A6NTBVqHI51CoxyJBUf9QXsv4+sDVgvqVWsQDD RoswrTGcvQxNRpRSEw4ateSoH47vdZ8PWjBeC9XQHtwBxYtSov2fw90M/6dX+45eNcbK 0hmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686273094; x=1688865094; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=DKO2MGlysYL+7+oQy8UdaapVjC7HViDdumwjMefUUFk=; b=CXMGvAzU4pgx9hI+RBz1w3BZD+0tCOmeXXPocZ80Xu4NKsK0DE/vUIphC0IZvFQUIP 25N4a+GiVAgpINVFqpd5uNSl+7fb6W3MJilMhw/og2ShovPIPxKxYMIC5bSnNDiTnfkE bWBy+za+YxoJq4Zlbp+/6MPGfNGEQdb7H7KmHlYv9aLGaSXpz/5Lpz0VlI4j8lW1JvrS wwJaxE1Voil/VNAjfp5t0CzY7T4N2CWJo7l+TaBPjXe4jeF+i4PVMK/k27VO6pO65c+k sS2LG01zD04hkrcLxiQjR/wExxFfyELpS/ChC9b8sjOrjv4JH3jhVO5TRELuI0U7ChEg BHzw== X-Gm-Message-State: AC+VfDwE/2SRTC3wIfaY6fXW30PFU82UZTqmhJ0Ov1z5vFtYixa+mO7A tSs87DdqsWn6q4/fpzSpiSy83A== X-Google-Smtp-Source: ACHHUZ43VqWz0kXy5mK3UV6S1YjsSZPeNy+eW38Rkn5m5MYDr07ka9TKGJWrHyw+7CpJWl7tn2JLMg== X-Received: by 2002:a05:620a:a02:b0:75e:ba6e:be65 with SMTP id i2-20020a05620a0a0200b0075eba6ebe65mr6769325qka.55.1686273094097; Thu, 08 Jun 2023 18:11:34 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id y63-20020a817d42000000b0056507de3d82sm273863ywc.104.2023.06.08.18.11.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:11:33 -0700 (PDT) Date: Thu, 8 Jun 2023 18:11:29 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 05/32] mm/filemap: allow pte_offset_map_lock() to fail In-Reply-To: Message-ID: <54607cf4-ddb6-7ef3-043-1d2de1a9a71@google.com> References: MIME-Version: 1.0 X-Stat-Signature: mjd3k5ec61jpz1ncy4gm4fprys6qabme X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 198A84000C X-Rspam-User: X-HE-Tag: 1686273094-536537 X-HE-Meta: U2FsdGVkX1+Iaa0zhVUy6gr2WIwifBS4vDDZuXyy6IDgueTHpgVg1kYvkejI6YMVt+TxL5JeXCoBkeQ5RrZtVm/2npso1/OXb44bKJ/TlQ6HG0W0xeir726zT77Z68Satkjj2WAoFBzsvY6ZjljtFdWHPDnSY68v8GaltSbCi6hG7b1krRH0DWC8Bgx1W3MPkrk8Lf9LCMnVnSglpAKrQTf1PqG1BoG+eSmHaWKh7txNhcs5jot9pjpTtP0qpYjLFPF2xsbb9eDmsWrQpFk8JVE4KnxaA7bYXvoUlbl5sYqSSoGaN8mZxF6owIkAzjp/hzLqVq5jvz0hQmuLMzIx3rSo0ZA2+Smz1PKy3w95kMKQym0uEURPubtwuYpHbysOUfk4No4V5UvObgY0Cb1dnkURJ8u4R2QVODnnbbXiHGimEr1k5OEbIsFjvtz8KZ3OVCyIuAWE/bRm1lQN0vqdpZ9+STuheepbC0lHH676xivgjyu1JYs6a2M4VbHi+dKAJke0RiEI3cQalqjxHeJuXHeWExojXw32rz0jPWjv/qHENKOz3eR8MnghYoAwU+ChaEVPBy6JAfqFaMLjrvrjC7NrKotCFTqzb/JaIQw+d3mjAOa/Ve2CcvmrypfFXR1G0ubHTq6fPMiyXfdTzJqHKwFXuQmbf7/S6GQ8jFYVjRLqyepmgeLYICBZc57zHlSZmfTWOo8zFsXs/WzKcYpRoITzWlI8k5k5yCjx+WIF7RnusQNi/PPvO1/reSA5ZI9Hz4TsjXcAiFKY5H+X8r/QvOG4zfy3k0yMYWmVSfO/6SLPDaor/ZnFiKCyNw7ff3Cm65GlrENkzOJRoUObJXi5k0/U6zMA7TpD03b6pcRfo8bUoenxKdkKG0QH+ec2hD32YxgNl278gijvvBFfZwlstvW7Lo2Iz+MMLmS9Mcy8zYFZP42TG8qfTaN2dNJl+AgmtQUxooSJmxk9mpSZKF1 LOnK7W67 4T7xv55OGPEUQszPAzUEBBlCJQJt5hnncOUaKIpnRODDF2uI8dZfTIv+ZUwus8hq/TmAkF29r/3xHcVzoh8iPZb7PVaMwhON1J72i46YF86p3/51u0cLobs1M9EMmc+aDCaprI/lVh468GKqVSIn5p7+E6UzJvGdcNBSkVtwJGdaCrnkvkwCGlh1T6bePpgGx5ZkUPdR9dOpyRChyuHPYqJqxFO9p8SC+L+SLDG3dFCoFGgWDvvu2a8mirM3Sslb9/3eszZdfEDhS1xPLMHbshU/KdVUtB8RvjAese81gmupUYmzohv6ifltD4KbCtFCkswDIBZYsxO2zUomZznv0lc0z+Dnf/qhB1Wua+qPfKGPLuurhWiMV0qwuPS71vV0aTj0m1w5OB5EJG+Ev1KwEq97cR8W8e1+g5mYYcmnDU7jB0ncZTT3l2Ug29w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: filemap_map_pages() allow pte_offset_map_lock() to fail; and remove the pmd_devmap_trans_unstable() check from filemap_map_pmd(), which can safely return to filemap_map_pages() and let pte_offset_map_lock() discover that. Signed-off-by: Hugh Dickins --- mm/filemap.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 28b42ee848a4..9e129ad43e0d 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3408,13 +3408,6 @@ static bool filemap_map_pmd(struct vm_fault *vmf, struct folio *folio, if (pmd_none(*vmf->pmd)) pmd_install(mm, vmf->pmd, &vmf->prealloc_pte); - /* See comment in handle_pte_fault() */ - if (pmd_devmap_trans_unstable(vmf->pmd)) { - folio_unlock(folio); - folio_put(folio); - return true; - } - return false; } @@ -3501,6 +3494,11 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, addr = vma->vm_start + ((start_pgoff - vma->vm_pgoff) << PAGE_SHIFT); vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, addr, &vmf->ptl); + if (!vmf->pte) { + folio_unlock(folio); + folio_put(folio); + goto out; + } do { again: page = folio_file_page(folio, xas.xa_index); From patchwork Fri Jun 9 01:12:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273067 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5248AC7EE29 for ; Fri, 9 Jun 2023 01:13:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E52308E0002; Thu, 8 Jun 2023 21:12:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E29038E0001; Thu, 8 Jun 2023 21:12:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CF1F88E0002; Thu, 8 Jun 2023 21:12:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C14C68E0001 for ; Thu, 8 Jun 2023 21:12:59 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 994DDA03A5 for ; Fri, 9 Jun 2023 01:12:59 +0000 (UTC) X-FDA: 80881435278.06.8AB4402 Received: from mail-yw1-f173.google.com (mail-yw1-f173.google.com [209.85.128.173]) by imf25.hostedemail.com (Postfix) with ESMTP id CB3B7A0008 for ; Fri, 9 Jun 2023 01:12:57 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="W/4m2uxI"; spf=pass (imf25.hostedemail.com: domain of hughd@google.com designates 209.85.128.173 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686273177; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zTm3eSBiFsuHpLmgEmMbxBID/kzy6WIASRGS/m4xNS4=; b=WJevkJd76u+8AskiAyoS92QuKu241Yl8yGNBV/GWNsHFP0rRjxmf/4IJ1AZzSwM8Fr1men +j3LICk3qBuxf3Ho/MslMjVPdgAjDqXSkKDNWg99QevOByNLMi2m97qWqZrTS+NaAqr49R pZAupT7qAVj6SALANDupDkG82Hnf53M= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686273177; a=rsa-sha256; cv=none; b=0fCTHgRNJzp0yqWBA12ji9fnE6rMa8czch1iu9KuXZjPSEl0b4f882qEh1Xk4yGvMPVeHU sYX4c8lMLt45FZTt7bFdQyn8NcnvE/MOHgpcGlWp8fHHDX9gz2qoC5J2CX7Jfs+MBHKye9 +w4Bhj+E8Jhl24auE4l12PxsHwU2fGU= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="W/4m2uxI"; spf=pass (imf25.hostedemail.com: domain of hughd@google.com designates 209.85.128.173 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f173.google.com with SMTP id 00721157ae682-565cdb77b01so10792497b3.0 for ; Thu, 08 Jun 2023 18:12:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686273177; x=1688865177; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=zTm3eSBiFsuHpLmgEmMbxBID/kzy6WIASRGS/m4xNS4=; b=W/4m2uxIgiBxYZeqFiaOJ+t56Xk1lzeX/0joWZYYIlyVr9x1rZQ0PVY6idfgJo3YAs J06WtHIbHcuFZFi7CTRbs4r/YWp12HHJP2GIpjC6Vtuxoe9P0OFgcqDXalAZfOXD7v9f PF1TUdN6QUVjLjHU9HZGO/hqyJB/REsEub0tjNaQLQmNIjXpYIuYk+iYoQFltIweg+DY h7yV6zg9eLJCboUD4lcwwTs6Aiti1k/7W+hJoajF2B88W4XFTVhUJC0bytvim02PYS13 jSNRaWmPHUADOzQcYjUwHqikvc9EsK2V2BuWiEGpR+wVTRiltwgLI4HXCzIB2BRua0U+ cdFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686273177; x=1688865177; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zTm3eSBiFsuHpLmgEmMbxBID/kzy6WIASRGS/m4xNS4=; b=OpTjI0NbYO3Te0JGCHXF0b8XywQ4PQhy8/k7BX4kMfHjaYkh5ZSeq8Kiu60oFFRS92 wbT4A+EMow0RQ0IsfdL0l0spoOz4zsDk+lmt0enF4jiI/Brxx1w2aJLG+aYWuMtLXZmR pQt/SAz/aFqy5z+tuHXu+sa+lmuIqDOmAOs4DyZuGCwsXONVuVkDs/9etsmHNBYutEhG dwJCV5wFfeA6aLtbGQ+X2RI33X1wRoASKteN1Q1l5o/kYVJl36oiz0ZaV7ANzBKfoHOs IuX9c/L4LWlul+9q8Sv8SbfX6JaOsKmwtX5XOtENPlebjjsI/Q1QQRpoXDJSW9ZZTb+k +lKw== X-Gm-Message-State: AC+VfDzGraJ0W0lC7Ltl3y28fy3+HKxUElGWZxTHjuiUN6wgJwcYybCZ MVvDVethR1HOIKCGX63EQrsqtg== X-Google-Smtp-Source: ACHHUZ7s2CfLundXAjg1vKJyhl3Rk4a+SVhLQYBVIqG94BbiAa+Lman4+SqaG4ekKQNDIkezzL6CKw== X-Received: by 2002:a25:b11e:0:b0:b8e:cb88:1b69 with SMTP id g30-20020a25b11e000000b00b8ecb881b69mr1064722ybj.34.1686273176726; Thu, 08 Jun 2023 18:12:56 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id b1-20020a252e41000000b00ba6ffc7ef35sm573643ybn.65.2023.06.08.18.12.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:12:56 -0700 (PDT) Date: Thu, 8 Jun 2023 18:12:52 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 06/32] mm/page_vma_mapped: delete bogosity in page_vma_mapped_walk() In-Reply-To: Message-ID: <87475a22-e59e-2d8b-d78a-df376d314bd@google.com> References: MIME-Version: 1.0 X-Rspamd-Queue-Id: CB3B7A0008 X-Rspam-User: X-Stat-Signature: yhmbt1hj5joq1i95zg68uwgnb4tb45qp X-Rspamd-Server: rspam03 X-HE-Tag: 1686273177-28197 X-HE-Meta: U2FsdGVkX1/1mdIJCYP87TuUCL0F4ZrZq5Lc7BU+amgiAeqyOBvuWjYm3VUWCIhFPl0zGT16xzn1kozSnS/9n/x8XNpxghZc80lxRVqWrWdsHaLnfJC7kI+NKNQYIkAp+4kJgbpVzOlk/oxTa354vAX7Vo/8MQszUgJ2rbdAHeMawECEoBCZEvlprbyPfsKrptekZXYEVAyv1GUR7fdOOBCjy0kG9vnaHTyhXJRYLpiTI6Rs2EZONsIvpGPrPe6/vQZ/omksJK2LYkyJdAQQFQXOuNw4m7ELriSRwYLucW/S/Z0zWzWKkkKDvmGZwdevMih/nTw9IVOIlGjPb2JFOIVHBdnlBxNcGTsERDXBFXrKzE7YlEd5jFzu5jQZEvPp/hIxnVl2OChMBxtLukknSol+P4vptxAHJ4hVS9pIO+YFJPS573Db9OjPJyeOZxHXbAjucZQYHWj6VjNvrUFARu4vgMc1tA9M/1P2uaTYXax1b17SGI0hA/eCtOTI3iqehoGG+3fRcI6sCrxwPdxzBrltJ7EdYMb8GyxlboQy0uQoGrU4YCljtw3vMDgAJjGqH2k06hVnXYQQnrfw/kgIqi10HrCu1a478DDDq+ePp3diotYePyivWa3XjA4yCJC15pMInTNVtl7PajgVR3uliwvXRcvQaOk53XGfBVS3zpFvzAyAzrn1sbbl3Q1FdYGafy/MCRy5Du2WcvGDFdxedjgCU2BvScNx9iwrjMhOWxpVbMtyhkaTaf8rqDPLUAOOX3ARkAvK+ONAHCKWcU2OYPAd75GGB3QqylD2Q1jqB+96l/DWCybBY2cSmxdraSiBn3qnGIXHlBdTVaVEyF6tN7MjRlwDzNsPDM8HiHe3ZziC03P28f6belpShkAhALzYik+wSzClENAD9WYGDew7yzLZgcwW5QDJuJR52F4TdEMzi19DwiFYu0FRocSC2I0HO9V3AVq5MI7tOJviTUI L/Ts/C2r FqvO6ws5SqlXi4yxxPy7CvWItAN5xWt+vzYfUF/x//ZLxzUnp8jy5yWfzPRMGsn9T/046cS7o6UE+Rlfa7BlU6SRGtWxRaIxfiUGR0RekoR3VCqc1wcU6upHDgkAFrmLKzJbBTrLVB8x5HUL0Q3Mu9SrquaV00wL5KE8D/1ZfLuSnbqCIw+cycforOHkwWwJOubvwkMVXvLADRK3rnTqHZs5v2opMzQ28Ptf/6zLFZzDSLJulgrX+Dubz6RdTsz2ZsIwxECgM0O5lzJ/bxsbhl70/kG88/Bk+z6InHsKjanJ15FwFeTXWmT7rg+36xWyNrDA6uemFbC89cflLf/BhCa5F+Bsk0WyeQA3WSmyc0l23mBYh+tfo48+mnOk+J4aGUkA/ZxAI+f4Z7lw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Revert commit a7a69d8ba88d ("mm/thp: another PVMW_SYNC fix in page_vma_mapped_walk()"): I was proud of that "Aha!" commit at the time, but in revisiting page_vma_mapped_walk() for pte_offset_map() failure, that block raised a doubt: and it now seems utterly bogus. The prior map_pte() has taken ptl unconditionally when PVMW_SYNC: I must have forgotten that when making the change. It did no harm, but could not have fixed a BUG or WARN, and is hard to reconcile with coming changes. Signed-off-by: Hugh Dickins --- mm/page_vma_mapped.c | 4 ---- 1 file changed, 4 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 64aff6718bdb..007dc7456f0e 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -275,10 +275,6 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) goto restart; } pvmw->pte++; - if ((pvmw->flags & PVMW_SYNC) && !pvmw->ptl) { - pvmw->ptl = pte_lockptr(mm, pvmw->pmd); - spin_lock(pvmw->ptl); - } } while (pte_none(*pvmw->pte)); if (!pvmw->ptl) { From patchwork Fri Jun 9 01:14:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273068 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B90F6C7EE29 for ; Fri, 9 Jun 2023 01:14:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 529DA8E0002; Thu, 8 Jun 2023 21:14:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 500E18E0001; Thu, 8 Jun 2023 21:14:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3C91D8E0002; Thu, 8 Jun 2023 21:14:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 300248E0001 for ; Thu, 8 Jun 2023 21:14:20 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id F22F0803A5 for ; Fri, 9 Jun 2023 01:14:19 +0000 (UTC) X-FDA: 80881438638.03.68798A5 Received: from mail-yw1-f172.google.com (mail-yw1-f172.google.com [209.85.128.172]) by imf20.hostedemail.com (Postfix) with ESMTP id 1E0E41C0002 for ; Fri, 9 Jun 2023 01:14:17 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=b0CpC9Xp; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of hughd@google.com designates 209.85.128.172 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686273258; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dk0QCHZ/ZrkXov8P0kyfxgscgrLXPV5n2YV5/OCWJso=; b=2MWRF1AMmXl5IqTTB6BwjLbYBLMhEjNvQ4Xp9GWlJqNmh55iRT0ktu+KX4u8LDevHKHbZy s0hubWaoFHCWFwoU7tECrRYcpQxzpZlwl6E1D2nJu/E2c0uezoBEr7AAblW5F3aBXaTINj VVNUxROjcBUYls6NZMRxxmBSAh77QqI= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=b0CpC9Xp; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of hughd@google.com designates 209.85.128.172 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686273258; a=rsa-sha256; cv=none; b=V/1nZSR1gd4L1iedqNkiUvpjlUX5x1aNcFO1QStLe1y6nN0TqtX7DxosMfNOfujU9UTarl hElG1LDSbTQZEwFXp85jTxDkxck8YG5homii9b83HJqHRqZLUv0Amvq4mLFPsYpVOidMTi oP23WDgbjWNsqpbDjnArKYJvmFY9kfw= Received: by mail-yw1-f172.google.com with SMTP id 00721157ae682-568af2f6454so11288877b3.1 for ; Thu, 08 Jun 2023 18:14:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686273257; x=1688865257; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=dk0QCHZ/ZrkXov8P0kyfxgscgrLXPV5n2YV5/OCWJso=; b=b0CpC9XpkwDMafp8znPdIUsufEoFF4TcEqSqtaBP/NRJK+TenjbuJzsvixBg7OeqD/ 2YfCQZOz9K140u4I0LV9p7i9HmUHhwPj5OjQEN4SArRt2Or8i+5jxZhLGecB/haRDICX vtLEyf8OOmoIO0wzxw5muMOyeWGS5tJSD8isNbt74/I3PyUiS3JXgFtf8p9995UjX66/ sVUXiIkxjAllgABbdRX2liHFwE/N9RUJ3gtPDEiB01DtZbdswHmgd6fCDnuNDxOd5cgB 2PYb4O3ns/eFa8oej3CUcEw8DbK3FH4hcK4a5VnPjgKE3o+d2ZfZbD5hqntDTaQ68DiO 87/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686273257; x=1688865257; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=dk0QCHZ/ZrkXov8P0kyfxgscgrLXPV5n2YV5/OCWJso=; b=fPU603f7paasB32QkXNbmPykBHjf3atJGA6sRZIir8Sltj3cuzECJvD4xVY5CMio0m psy+HWmNUr0ODnkFROPE++3LJSSEbR3uIBnLGMBof2rA4sFXI9fOuKiQdUqc2VjJCuWr NaMxrPnYrmbqLyITkkRVGDqLRVOupDGXTZzzeZTL4UZizK4mTzqqMGgqoVMLFwopL7k4 tqeyPDwq1x6gb9XwecYCf0/AzmjyY0dLAohBOpx/UP029U+biDsQpDlNZaA5z5VKF6Yx yjXILmLQeMU8nb00RrgTi8OACBbpx7DvygnAaBoiB8MbFHL2OcRafCzs3/9lgMueP3jp zvJA== X-Gm-Message-State: AC+VfDyp626PWEfd20DgTj40MVSKXDxcyFt0TBoLt5zAwSEHgm2CD+6x tCW+UAYV2AFITbNQPamNQE6LLg== X-Google-Smtp-Source: ACHHUZ6G5/QqeKVeqB1vcKg8Kl9KAE//8sPZoP2ai/tJpfMIrL/rfl6x9oi+hpyk1yDqLu4iwiQrEA== X-Received: by 2002:a0d:d6c8:0:b0:549:2623:6f65 with SMTP id y191-20020a0dd6c8000000b0054926236f65mr1346506ywd.33.1686273257036; Thu, 08 Jun 2023 18:14:17 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id t66-20020a815f45000000b0054fba955474sm301207ywb.17.2023.06.08.18.14.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:14:16 -0700 (PDT) Date: Thu, 8 Jun 2023 18:14:12 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 07/32] mm/page_vma_mapped: reformat map_pte() with less indentation In-Reply-To: Message-ID: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 1E0E41C0002 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: m1okr8poj794hcot3p18cm4btdbdi1s3 X-HE-Tag: 1686273257-872995 X-HE-Meta: U2FsdGVkX18uFpp/4lexkYDwMnVIWKpRLW6HBD7N1+tfXYOC/npaNyoMU7O12upKFJnvsIx89vg2enxr2sOOLvUnYyLAxqpR1kpUHIjNXlqeEEw9WzRCOwrQhhCQYe7LjwdwG7yAumWVvF7vMCHdnzP4DqdQ0WpqitNhjLebpAKMB50VSnrA8YRh6CcMGG0FuCGV5A+HogwAapCgns8exaxpELb4BLl26r7LtYjmLP9sI7PypN3CNvwVj2I7CLRiAzHlMQ4eR3yj0Nxx8XeX710/H2qkxJz3vguT5EJ2NJtUEq8HFQ0aHSanBrdPiEA2ogtkoXeSKPphh9N9IPJbgOCQW5afkBaUvN7thWxTRXQOLneMjeCc7CoOTuYcHP8KDvO0a6dYN1YlHSwi75Esx2pwS+rxE/dAwfPlBy1h2Z2Ti3YnWal3AgmJFOYl3JKTDkLRRTmaZ196n6/CjvS/Pa5rzwEwUnnURbzwsOs8bd1KXrhUZHR2tzojOhWZ++bkPut7UXwBh5iePvMhBBgSB5ogmF60Ts+scT/pWtvEUsyFsym3rpJ3mEAHLZ+oV40cP3o4wb6RTLKxsYgjEBZRMM2wffTCyLIjf4AyFG7mIgEndNjO5Cup0Zfxsl6qF5ZWzYH+QuY+I8TrJUheu3GLgB8WTQVpAgew/QyPME6+hW30ep5yeaBEWxMKs+9c+Th2bsPNK3ZrdYLiIljo27K8ZQg2ZBBsb5qAaVvaNxVzTrvxH1qYyrDozmELtts20X4YjFQ+/KA7JAUXvPnjofgoWZoEf2GCo8ALf2vHecY3djg5QvIcneC6IzsCJwFp96TaZGeBtfMXqxmop6d8nH+4xtglVtGI5qYnB9+tKNAR8JF2oW7419AtzGV8BIrSQ1oarJui5IPbyiywVbllliYKRwKFE7oFmv8Ha0De2FieOM0orzCXrh9Kn7MI49OfgJs91bscfbgYXsdSmSRTuA/ gclqhb6a MpQJkm4d2O/WEzUKR4NQZ7vCVdG2RSluHru9oPtzlA0mz5PO9sFjPDvLU962cMuv+2/06MUKd60Y8hp20o2KxcOhxD7M50Pqv/HQ7ayKwAqbzrhZntqVS+MjQqxDoVIhoHcLdzLJM63+fbv81dA3Mj4PZE3VeXOvkzm/M7sfvfqJDyWnpgku/Ccz8HinwQoWbmap5y9P+ZyD9xnJgfVipruthePJYZfvkocGQc8Ots2KZRog3PpZ5bZXPFqPRmre5tQt0PW4aj93kcLOr2xWCxbV4RRroqCXbsprsjy/TUNXXSfsBwxvJ9BF7jmpNda4mbHkJ/RY4Lpn8eVAobjsgN6qoa3CU+r/VJ+2EycoRUtgeBFI3ugebxHc9lgxzDHTVOI2RVlBpe8yNQ5PWSWoKJ8cT+WXEfSkqsEK++euyQXH75MCIJi8t/tvwuw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: No functional change here, but adjust the format of map_pte() so that the following commit will be easier to read: separate out the PVMW_SYNC case first, and remove two levels of indentation from the ZONE_DEVICE case. Signed-off-by: Hugh Dickins --- mm/page_vma_mapped.c | 65 +++++++++++++++++++++++--------------------- 1 file changed, 34 insertions(+), 31 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 007dc7456f0e..947dc7491815 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -15,38 +15,41 @@ static inline bool not_found(struct page_vma_mapped_walk *pvmw) static bool map_pte(struct page_vma_mapped_walk *pvmw) { - pvmw->pte = pte_offset_map(pvmw->pmd, pvmw->address); - if (!(pvmw->flags & PVMW_SYNC)) { - if (pvmw->flags & PVMW_MIGRATION) { - if (!is_swap_pte(*pvmw->pte)) - return false; - } else { - /* - * We get here when we are trying to unmap a private - * device page from the process address space. Such - * page is not CPU accessible and thus is mapped as - * a special swap entry, nonetheless it still does - * count as a valid regular mapping for the page (and - * is accounted as such in page maps count). - * - * So handle this special case as if it was a normal - * page mapping ie lock CPU page table and returns - * true. - * - * For more details on device private memory see HMM - * (include/linux/hmm.h or mm/hmm.c). - */ - if (is_swap_pte(*pvmw->pte)) { - swp_entry_t entry; + if (pvmw->flags & PVMW_SYNC) { + /* Use the stricter lookup */ + pvmw->pte = pte_offset_map_lock(pvmw->vma->vm_mm, pvmw->pmd, + pvmw->address, &pvmw->ptl); + return true; + } - /* Handle un-addressable ZONE_DEVICE memory */ - entry = pte_to_swp_entry(*pvmw->pte); - if (!is_device_private_entry(entry) && - !is_device_exclusive_entry(entry)) - return false; - } else if (!pte_present(*pvmw->pte)) - return false; - } + pvmw->pte = pte_offset_map(pvmw->pmd, pvmw->address); + if (pvmw->flags & PVMW_MIGRATION) { + if (!is_swap_pte(*pvmw->pte)) + return false; + } else if (is_swap_pte(*pvmw->pte)) { + swp_entry_t entry; + /* + * Handle un-addressable ZONE_DEVICE memory. + * + * We get here when we are trying to unmap a private + * device page from the process address space. Such + * page is not CPU accessible and thus is mapped as + * a special swap entry, nonetheless it still does + * count as a valid regular mapping for the page + * (and is accounted as such in page maps count). + * + * So handle this special case as if it was a normal + * page mapping ie lock CPU page table and return true. + * + * For more details on device private memory see HMM + * (include/linux/hmm.h or mm/hmm.c). + */ + entry = pte_to_swp_entry(*pvmw->pte); + if (!is_device_private_entry(entry) && + !is_device_exclusive_entry(entry)) + return false; + } else if (!pte_present(*pvmw->pte)) { + return false; } pvmw->ptl = pte_lockptr(pvmw->vma->vm_mm, pvmw->pmd); spin_lock(pvmw->ptl); From patchwork Fri Jun 9 01:15:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273071 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6688AC7EE25 for ; Fri, 9 Jun 2023 01:15:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F3B2C8E0002; Thu, 8 Jun 2023 21:15:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F12248E0001; Thu, 8 Jun 2023 21:15:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DDB5B8E0002; Thu, 8 Jun 2023 21:15:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id CEC988E0001 for ; Thu, 8 Jun 2023 21:15:51 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id CBF13C031B for ; Fri, 9 Jun 2023 01:15:50 +0000 (UTC) X-FDA: 80881442460.10.C471257 Received: from mail-qt1-f179.google.com (mail-qt1-f179.google.com [209.85.160.179]) by imf23.hostedemail.com (Postfix) with ESMTP id 037E0140015 for ; Fri, 9 Jun 2023 01:15:48 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=qGqUUKrU; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of hughd@google.com designates 209.85.160.179 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686273349; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9VYOFAJpcVHpYrrS1TQZcIVdYEqSjI0GrwCgA5USRTU=; b=CrQtUuwJaQedCm9e1Rw4oLBWnzISNEOSTAK82qtFTXUSfUTc+I/MDh06JUvlm+DzLWeHqf IWzUTocw9fRmLwT8eCXTEkvWVSC5oksfW2td2kuKVFw6jRRPW4FQxCSeOh/Ox+R2CAHRyx SGG8JYLtHP8dhq3KAq9SiEBZcIjAAuM= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=qGqUUKrU; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of hughd@google.com designates 209.85.160.179 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686273349; a=rsa-sha256; cv=none; b=FDjUN+bai+sHNNunc1tV+ml0IxpLqzoo8a+dXtKHrczrTPjqYmUg9OKvTF+I4Bt4CclqcY O1PY020qHGlsmuXst9F3PqC87aI5cmG09bCoYphAsCcHSYcNfbjE90yJZ4fmDYLgve7ZsI Fenjm6gdiw//0oo4sv9k1Cvb6MDs6gc= Received: by mail-qt1-f179.google.com with SMTP id d75a77b69052e-3f8008cb772so10534141cf.3 for ; Thu, 08 Jun 2023 18:15:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686273348; x=1688865348; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=9VYOFAJpcVHpYrrS1TQZcIVdYEqSjI0GrwCgA5USRTU=; b=qGqUUKrUhXZqdfIpdRpru1enURAlAM37w3JnFCBdqcN5NOVKva/5ZB+Wx1AncWuaoB 4NOGT/sluCqCWe9k46AbMdrfO/y7S/uoPCSIv3YcbyPtfcHQLmMhf87/VSB5iKcTyY2z DvH2xbXgY53bvpfsWKoRD42G3UAVdO/UJ+NkrWhrUDvDf4DfgkGc7vx+86uzLhaB2Vdl OBAdZ3Qvm5QESmYlbn3dnRPJ/xpFtDzek+36j2PNxi4y2bmdWvVJycdWxpou775822Zd vv/wKkp98uy9cbO4mRL/ywca8CgLJugp1BRNkerDGGNXhj+j6x2eNtUbygfDvk9GNFDY 1+3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686273348; x=1688865348; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9VYOFAJpcVHpYrrS1TQZcIVdYEqSjI0GrwCgA5USRTU=; b=UIbx9YWmuHg4tRsZ2CwYCdeRKYhgTs/9NsvcJ7yBwqSQxv8w7+L1QEjOqe4MBOSN24 EAFaSLVIekglHrCqQ4pXWzO/yx5MU8HbJ5eugG+VcJG+sY2+t/fhdngIvmv9j0TKqHSm sKJFCwhEDc1Dhp7FJvO3aNz1HeargOLvTkI0po4c8GvtclCIehAvsJEcex8Iw+ciJl3X ca2dPvOVvczoo7Lsgg7ppKzOEl3mangz6MpL55pJYmk+B3/maywqaWRNzmn2RKBbaYqp gKm3MRfUUo0lXbOC3VmGnXXsIMt1AA6fQPulYArzhKs2slrJHYkeVtnUml8MWR9xk/Sm tJNw== X-Gm-Message-State: AC+VfDwnX77syhohI3nqFyvya5nMXzx7+S71KK3lpI452Yne70m2uDEW VhNIDf/KLRWSW9fDioap6itQfQ== X-Google-Smtp-Source: ACHHUZ5RPLaTpoCJzmmgtuS8+EduaIDi339U1d6YVPQERnp7XZzVYgUh9cqEFrnN8L2JonZ6XbEg1g== X-Received: by 2002:ac8:57d2:0:b0:3f6:9a18:e67c with SMTP id w18-20020ac857d2000000b003f69a18e67cmr42699qta.66.1686273348055; Thu, 08 Jun 2023 18:15:48 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id u15-20020a25840f000000b00b9e2ef25f1asm583095ybk.44.2023.06.08.18.15.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:15:47 -0700 (PDT) Date: Thu, 8 Jun 2023 18:15:43 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 08/32] mm/page_vma_mapped: pte_offset_map_nolock() not pte_lockptr() In-Reply-To: Message-ID: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 037E0140015 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: osfwnknrqjzc11y6n8g7ui7mrmufbpkp X-HE-Tag: 1686273348-207053 X-HE-Meta: U2FsdGVkX1/wEfW3DZdcjFK5HwtB+pAO964SCfjfMZFZH3vRRaPDNU8orwQEYRAfHUEAhIIVcU4FhI5fuwKVuXAjmT1sAe6b+yms8iO9aosHM+Y5lIIQqxal0rk8hi2kvLKKUiGCm2rRWAQdsOlhDSk1BI2IfDTkUMwmES5PugJIQrLewZQiAh/uU3Ezhxj2Rqe1DfTXBADlAuIJfkp+BfzQVB+vjI/IY3cHCkcxNaksjrIPkf2zUQg15tA/SdnTBw8PbYdMAAG2AFHoTAZvABTSMxdHhI0HuKki7jybT4qOq3DmUMXP4nn8T1+A0eR7bDi82Q54lkzUTxdBcKvQksgCb3Ne5EtYOq50fr9BiY2PA5i6Jt42pcGv/lL+zyE7t/vDWo9LRTuDj6JqH8fmkic/mixmvWoXrIpY/txo2REkoTpHNiBLSiIg9rU/xQIa139jjWLApFs73daH5Q6/O2yZYx5AKHIBLeNGuo6/YLbV8AqypD+0Z90oGUXIECtCflLLlCsgRLpitOuivkIncShiFqGytmMPzGWa4K1mAoweWTS7Z3MwFh6NqhKzAcTI69bOUpE8Cin5zzAfbi0MFyX6rcJuVrF02ItMNURvS2giQo7pqEOQPh4BcvN7I0W921oK0ukujJuh5A4zxmLdfnNGC6CfQ6fzY0VjiRLes/HrP1XdYKy/F2U3/DjP6ExF5QKu+qjmzsE6SkEw2WZf3qmJLPnELUgXPGXlrqKjKQueTv/rsfggSg6oFUCxSX4CbVFfXUeY5q4ad6hHtUwobWcyWGeGlPDyVopaB/4Ivc5yuDhUBgtJ6i1ogpLwUBxU/wS1ctTI4UAY8rHskgeW8EE6n0u8XOa06Mpyze/vYCLofi9hIP7/WeYqRA+i0UCIe2QmltsKv48Gi/ie+YBXDbL202jsL7RQYZEry7io4ni3BiBpagfPV7xe3AQkPoabvJXnwlnFxQnMlzmbjeE rDe78KAJ yhODVIBgXjndrQgGu/swy5MAS9Xl9narHtG2Uv0zYGwDWms9TPqhW2ZjY9sVUjsmqecUXdt2iMZb9rfEvWrAWnqRedx2hMeiXBFFL2+Eo5KTlNZlWWN7HGXH9UYge7VKCWHLR80WcUuoQpGkOwR0SJAaAJDH/x72sLSWC85douPKnEwJ3JnUwJKX1kZl2fhLm59ISGUaoDYfV+Jb+PffhSPVbFEOrVXtX/PJP02tDDry5TDf09f397IJG/AOORW9hfvxhIcN70wn8HrOqEIzQfJevP6r1ZrYMn9bj6TjsWM1tRsXZR/wL7iUvJQpAyq4WTZpXce+nUX4sKyr+lgDIMfn/TQPYlRxlvTyEMP02MaJF9fmg14tmXPMB0WwFZL3aIUIjP/0y9rKw+ekZ9Xbmsoay/b2Lpm3T9TSZZyP/3oHtbNWBNUjFQjTNpw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: map_pte() use pte_offset_map_nolock(), to make sure of the ptl belonging to pte, even if pmd entry is then changed racily: page_vma_mapped_walk() use that instead of getting pte_lockptr() later, or restart if map_pte() found no page table. Signed-off-by: Hugh Dickins --- mm/page_vma_mapped.c | 28 ++++++++++++++++++++++------ 1 file changed, 22 insertions(+), 6 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 947dc7491815..2af734274073 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -13,16 +13,28 @@ static inline bool not_found(struct page_vma_mapped_walk *pvmw) return false; } -static bool map_pte(struct page_vma_mapped_walk *pvmw) +static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) { if (pvmw->flags & PVMW_SYNC) { /* Use the stricter lookup */ pvmw->pte = pte_offset_map_lock(pvmw->vma->vm_mm, pvmw->pmd, pvmw->address, &pvmw->ptl); - return true; + *ptlp = pvmw->ptl; + return !!pvmw->pte; } - pvmw->pte = pte_offset_map(pvmw->pmd, pvmw->address); + /* + * It is important to return the ptl corresponding to pte, + * in case *pvmw->pmd changes underneath us; so we need to + * return it even when choosing not to lock, in case caller + * proceeds to loop over next ptes, and finds a match later. + * Though, in most cases, page lock already protects this. + */ + pvmw->pte = pte_offset_map_nolock(pvmw->vma->vm_mm, pvmw->pmd, + pvmw->address, ptlp); + if (!pvmw->pte) + return false; + if (pvmw->flags & PVMW_MIGRATION) { if (!is_swap_pte(*pvmw->pte)) return false; @@ -51,7 +63,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw) } else if (!pte_present(*pvmw->pte)) { return false; } - pvmw->ptl = pte_lockptr(pvmw->vma->vm_mm, pvmw->pmd); + pvmw->ptl = *ptlp; spin_lock(pvmw->ptl); return true; } @@ -156,6 +168,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) struct vm_area_struct *vma = pvmw->vma; struct mm_struct *mm = vma->vm_mm; unsigned long end; + spinlock_t *ptl; pgd_t *pgd; p4d_t *p4d; pud_t *pud; @@ -257,8 +270,11 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) step_forward(pvmw, PMD_SIZE); continue; } - if (!map_pte(pvmw)) + if (!map_pte(pvmw, &ptl)) { + if (!pvmw->pte) + goto restart; goto next_pte; + } this_pte: if (check_pte(pvmw)) return true; @@ -281,7 +297,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) } while (pte_none(*pvmw->pte)); if (!pvmw->ptl) { - pvmw->ptl = pte_lockptr(mm, pvmw->pmd); + pvmw->ptl = ptl; spin_lock(pvmw->ptl); } goto this_pte; From patchwork Fri Jun 9 01:17:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273072 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C917C7EE29 for ; Fri, 9 Jun 2023 01:17:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E77A58E0002; Thu, 8 Jun 2023 21:17:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E27A18E0001; Thu, 8 Jun 2023 21:17:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CC8048E0002; Thu, 8 Jun 2023 21:17:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id BC8268E0001 for ; Thu, 8 Jun 2023 21:17:33 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 945591202B7 for ; Fri, 9 Jun 2023 01:17:33 +0000 (UTC) X-FDA: 80881446786.12.3599FDA Received: from mail-yw1-f170.google.com (mail-yw1-f170.google.com [209.85.128.170]) by imf05.hostedemail.com (Postfix) with ESMTP id C8C11100005 for ; Fri, 9 Jun 2023 01:17:31 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="hfFZS/To"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf05.hostedemail.com: domain of hughd@google.com designates 209.85.128.170 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686273451; a=rsa-sha256; cv=none; b=XWOmGOh2bp1XzYCtB7Ah1hx6iiZpD28lalwbQnhWRdX5pFNF9y5P44ZAyQ57zD0X0s9Fl7 FTOD3sRfqZDHtHiMxtxrUjfbLfbF6XUv8s5L5WqOME3pwoDZbpAY1gKKY78sdsNfOlkTU1 2Y3mZLkB7BqcRAFZbt0LAti9FcVZ4Lw= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="hfFZS/To"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf05.hostedemail.com: domain of hughd@google.com designates 209.85.128.170 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686273451; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YGxXR2sylNgXOWjswYon8LmdFOYpbwThTVM1QZPxixo=; b=IxgM6bSP3cZQSF3DryuNK9BFuiv3/zs6Chrp3+ZD0IgyjlaeV4jsLnCt7ZMzJBUEzBjy1V 4SNgirXvuBIzXG0ewVWqXC91hHVfGoYB91J0isEkuAkujkvL2nKdEpvdAuGLYjRA78jMCH ygbN6DI10H6agw3IXOgV0LiGEEC8Ht8= Received: by mail-yw1-f170.google.com with SMTP id 00721157ae682-569fc874498so12103617b3.1 for ; Thu, 08 Jun 2023 18:17:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686273451; x=1688865451; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=YGxXR2sylNgXOWjswYon8LmdFOYpbwThTVM1QZPxixo=; b=hfFZS/To38y/kHK62NcQe3NeUtPQadIbTbMGv9CwCwnaOm3+ubft1ovMBDYwgIcJHN maH31bmCKIXTcoH2mw+7AlwbT//z013fnJricqAgo1Y8wR2cLWz5jidfA/WWWdNn++DD eExfA1ytyJkqY/La00m6AnHJyexbZrQSpOFK5OQ6AFOk6cyYk4BIHtO7IYbubUELYE5/ t2x0w817NAACm7JQwyi+uQozGB37VzsQ0dIb5qTadGN0KMsSpWs2GUyHK1Uhm4UV4EVa KWOR3EiG1vfrWrdRdz5JYnJcT4eEEx83Gbf/GYzP39cr1/xlfqJ/0ZZkApmwpKDFSRuJ KEIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686273451; x=1688865451; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=YGxXR2sylNgXOWjswYon8LmdFOYpbwThTVM1QZPxixo=; b=JNmJ1X1Do0dRHkA4wR1KB2HoFUlmqsnkpu4iQDLIrl0uTsx/ZW/vtwR2PJuXvV92MC ChGHf1RmFlnxv4CIO4JqW+jai5+yutE5j2w2Dhi9rjDQr6rXvo4NdKnaBrfvhbwZ0n1J QHuXlUZFQ/7kg33YlxML5QBtJLQGTKx4kDwIyop58Om+MU8uJxqg096lgOkvZho2ELmO yuXjdjwb3MGoXpuuNZBUrVs6oFTgNTnniJDVvqhjRD+josWb8uZrQ2QtZiI7ODRTGrfz zpGxd9gPVgpULGothVvZVafSaavIFRIzH1UPsLo8aZyZJFQgUmYFUS5F9nK1Hs01EueJ P8PQ== X-Gm-Message-State: AC+VfDyNaf0kxdvlSpfk/zM5WY2FNSgjfVtOSFGNrh/2V0i9s2u5ET8q zfQGYHXD1m8sdsheTe89i/gpxQ== X-Google-Smtp-Source: ACHHUZ684nq9AHXRq1UVfeo4n1qoOEQQWARDc3yoeKJKitFHVG0vbG1oJERSIO9SpyMcUhVMBrYK3g== X-Received: by 2002:a81:6cd5:0:b0:564:c4db:631e with SMTP id h204-20020a816cd5000000b00564c4db631emr1257858ywc.5.1686273450756; Thu, 08 Jun 2023 18:17:30 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id p131-20020a817489000000b00560beb1c97bsm287394ywc.97.2023.06.08.18.17.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:17:30 -0700 (PDT) Date: Thu, 8 Jun 2023 18:17:26 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 09/32] mm/pagewalkers: ACTION_AGAIN if pte_offset_map_lock() fails In-Reply-To: Message-ID: References: MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: C8C11100005 X-Stat-Signature: wiz7ipamtzjqjda73gjszgdhck8c8t3y X-HE-Tag: 1686273451-999198 X-HE-Meta: U2FsdGVkX19TLZKvp340vPcYg0NdUjofadvbGjmiGsEN7F7xMfSYo3gxk2FlOalYJUsJAJgFnHGuoyTX0+KH5iBZntu7CPhTw+luFfi0BCrCZqDHZKe5nmmiT0JKgIKkPkk0XJV6Neg9v33DRrJti62jmrzhDBDoD7eIxXOGU5SqIYPES29Da8rUgFSJcaD8XKROk8HjEvpAq7iLY3q94LjEMyCubj+MVYUWEy05JA7SKKPjwfDAutlCdXA+eTY6aChQa1gwZRK5BeHVmIzLuc9c/470jXGj0Fvm4X6+4ZQEDeojvxU4yVqpjQcNVUA/Ws3acZTKDLGALv4TD0G2HrqKCP0YFfO7hhaSKLeGzCMiW4EfUf5aX93HpgLM6oPpp8Vwm+4zg2i7Otlup32fYNHm+dZfAAAS1GCEWDt48LgW0MAH2/fELweYTpYio+4TenzqRmjfuKHQnxqVwxDKMWik0ivCtVk/z+8cZrSzmtnxW9+cwIutA96ENPJwPDyRyRbaRCATTs/hHpVRve6ikHblO8oVJQxUqA8RnCEuL3Bvc6UrxhcXUK9OZSOAhPUJoVEuWmQzUierTnH84NuOAQx3T0tq0RNKMwzjwP1/lfXCkru6PpfOK0UREZQTgrpaeiU6qNAIVpv/lwFwkXONJ8Y1isxxDQw21cm9/1RYZhuEirb4sVa3N7lm4bRq0mxU9tJ5qx6e8A9ww6WV1BJkQX44HqCjDXHUbkCQV/aCqGHFyIWd8rdIZWhr0FKNnBqMMAFQBXQ3WODRJ7KJz2ii26wKKc3S8Ex24NRM65cYAboOga3uFjdkpvWtsvzXMSUlxD9IVH83yggqlf3YncUxkfFkJQqVWcqiS/TYkIQj6U3ZbDOcd5weBPWhA//clutRVpWHjVPH8a1pH8BWnUJuSCa3aylqVoIdeQqWb4s8w3a6g+H2OroE4HRSABrYrZ8SNqsDPW2Kcz2d/Vlgvja 71xtePYr r96j48gFm8IhyLaQldt2Y+5ZO0hqbzBGpLzUQqxU3/KnbDHbOM6U/rdp4Jus7Sbp1GQZk3+o3EJZGbaW41ZHfjGIria/WnlCRW/suXQZlnvkGhELepRIbu0Tt91L0k/szMTnpWWIPbxjG5e1xs9bNLgVwTAytZm5D4c72/TU//brYfirS0HmGESeca9VBN64OTSgwlD9fRJzrRcGh7JF5MYm1Uh8T4KZcoxZ9ztjsmIlaiz0Pt/92Q6XSQnK1kREgaDfFy4pDpHUFR4TNhLDoLtTpj1lHSQqZD7VxxJzI7mtTdrV8rhfzLMsv5C9SXIOcKO/KbZPaTOcy4j697bQjOjCDWZDCeSbXUO2AcgW8wUtEXq69141Tq87mFZahNtC0SsmhvtLvc9kn1A96BQ2OH05jBBUX0joDLmM2kwapjm6t/b18whyt0vnPM+Ux/t26tna7CU4iZiZy8T2SLWQhINpM5xTjDOHmPi24fl/s15cYH7E= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Simple walk_page_range() users should set ACTION_AGAIN to retry when pte_offset_map_lock() fails. No need to check pmd_trans_unstable(): that was precisely to avoid the possiblity of calling pte_offset_map() on a racily removed or inserted THP entry, but such cases are now safely handled inside it. Likewise there is no need to check pmd_none() or pmd_bad() before calling it. Signed-off-by: Hugh Dickins Reviewed-by: SeongJae Park for mm/damon part --- fs/proc/task_mmu.c | 32 ++++++++++++++++---------------- mm/damon/vaddr.c | 12 ++++++++---- mm/mempolicy.c | 7 ++++--- mm/mincore.c | 9 ++++----- mm/mlock.c | 4 ++++ 5 files changed, 36 insertions(+), 28 deletions(-) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 420510f6a545..dba5052ce09b 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -631,14 +631,11 @@ static int smaps_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, goto out; } - if (pmd_trans_unstable(pmd)) - goto out; - /* - * The mmap_lock held all the way back in m_start() is what - * keeps khugepaged out of here and from collapsing things - * in here. - */ pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } for (; addr != end; pte++, addr += PAGE_SIZE) smaps_pte_entry(pte, addr, walk); pte_unmap_unlock(pte - 1, ptl); @@ -1191,10 +1188,11 @@ static int clear_refs_pte_range(pmd_t *pmd, unsigned long addr, return 0; } - if (pmd_trans_unstable(pmd)) - return 0; - pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } for (; addr != end; pte++, addr += PAGE_SIZE) { ptent = *pte; @@ -1538,9 +1536,6 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end, spin_unlock(ptl); return err; } - - if (pmd_trans_unstable(pmdp)) - return 0; #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ /* @@ -1548,6 +1543,10 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end, * goes beyond vma->vm_end. */ orig_pte = pte = pte_offset_map_lock(walk->mm, pmdp, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return err; + } for (; addr < end; pte++, addr += PAGE_SIZE) { pagemap_entry_t pme; @@ -1887,11 +1886,12 @@ static int gather_pte_stats(pmd_t *pmd, unsigned long addr, spin_unlock(ptl); return 0; } - - if (pmd_trans_unstable(pmd)) - return 0; #endif orig_pte = pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } do { struct page *page = can_gather_numa_stats(*pte, vma, addr); if (!page) diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c index 1fec16d7263e..b8762ff15c3c 100644 --- a/mm/damon/vaddr.c +++ b/mm/damon/vaddr.c @@ -318,9 +318,11 @@ static int damon_mkold_pmd_entry(pmd_t *pmd, unsigned long addr, spin_unlock(ptl); } - if (pmd_none(*pmd) || unlikely(pmd_bad(*pmd))) - return 0; pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } if (!pte_present(*pte)) goto out; damon_ptep_mkold(pte, walk->mm, addr); @@ -464,9 +466,11 @@ static int damon_young_pmd_entry(pmd_t *pmd, unsigned long addr, regular_page: #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ - if (pmd_none(*pmd) || unlikely(pmd_bad(*pmd))) - return -EINVAL; pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } if (!pte_present(*pte)) goto out; folio = damon_get_folio(pte_pfn(*pte)); diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 1756389a0609..4d0bcf6f0d52 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -514,10 +514,11 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, if (ptl) return queue_folios_pmd(pmd, ptl, addr, end, walk); - if (pmd_trans_unstable(pmd)) - return 0; - mapped_pte = pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } for (; addr != end; pte++, addr += PAGE_SIZE) { if (!pte_present(*pte)) continue; diff --git a/mm/mincore.c b/mm/mincore.c index 2d5be013a25a..f33f6a0b1ded 100644 --- a/mm/mincore.c +++ b/mm/mincore.c @@ -113,12 +113,11 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, goto out; } - if (pmd_trans_unstable(pmd)) { - __mincore_unmapped_range(addr, end, vma, vec); - goto out; - } - ptep = pte_offset_map_lock(walk->mm, pmd, addr, &ptl); + if (!ptep) { + walk->action = ACTION_AGAIN; + return 0; + } for (; addr != end; ptep++, addr += PAGE_SIZE) { pte_t pte = *ptep; diff --git a/mm/mlock.c b/mm/mlock.c index 40b43f8740df..9f2b1173b1b1 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -329,6 +329,10 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, } start_pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + if (!start_pte) { + walk->action = ACTION_AGAIN; + return 0; + } for (pte = start_pte; addr != end; pte++, addr += PAGE_SIZE) { if (!pte_present(*pte)) continue; From patchwork Fri Jun 9 01:18:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273073 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1FCBC7EE29 for ; Fri, 9 Jun 2023 01:18:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 662B28E0002; Thu, 8 Jun 2023 21:18:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5E9DF8E0001; Thu, 8 Jun 2023 21:18:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4641A8E0002; Thu, 8 Jun 2023 21:18:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 36D7C8E0001 for ; Thu, 8 Jun 2023 21:18:58 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 00561A03A1 for ; Fri, 9 Jun 2023 01:18:56 +0000 (UTC) X-FDA: 80881450272.18.BA5857F Received: from mail-yb1-f177.google.com (mail-yb1-f177.google.com [209.85.219.177]) by imf06.hostedemail.com (Postfix) with ESMTP id 33E2F18000B for ; Fri, 9 Jun 2023 01:18:55 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=ngXLVdTp; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of hughd@google.com designates 209.85.219.177 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686273535; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=s/wneZk+WXTrjMA3fZuua9PXiv3fUuHnEiWoD80tgK8=; b=6eXUz20BV4BC3to9kLOyiNzLOEsw5FrffPaUywGt9JJcMru86qxjkL2SIU0qbxTIlxv0jb kJmfanHFPkij/CLpBBS8K9VOdr20HLvVNQSAx7cpTsNeEL9KcP07admtToImkxgncFzgdx aHL925n9JR7fTQqbIVECFZUM1UjR9+k= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=ngXLVdTp; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of hughd@google.com designates 209.85.219.177 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686273535; a=rsa-sha256; cv=none; b=fpuiH7ttDwU9Wa+gCuJiFjZ3KjHPxM1BSzPURi99pmBLb5PHoD98HiiY6la3secOtwIQLp UN/LNmZiQMngR5mAtN8j17+BSs3kAsu72HGIl7bdxPOZZu3y4ugwn4acrgJAQ2jEEq1EXJ RnX0AFSCNaCbYp9eiO6wqCNpx0uuyl0= Received: by mail-yb1-f177.google.com with SMTP id 3f1490d57ef6-bad1ae90c2eso1274939276.2 for ; Thu, 08 Jun 2023 18:18:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686273534; x=1688865534; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=s/wneZk+WXTrjMA3fZuua9PXiv3fUuHnEiWoD80tgK8=; b=ngXLVdTpVyrnOmcx9sqd11fDN/DpMfDjRFc20sS67KuKXUBPfJVN7b54WivfwCCqcm 2w8p0+WuKxMdG7YLSaNER90dnOHjPSlvPspYoe5kIeMhFky06+Enxr+OKG3u1lmvAOqa C+PVbym1M2SnYvLxMqSZg33YxKhOPm3FgR9jzQ+MnLfZGFUC2OJbbjuScpkVkepg/TEp 276YbysaG6RpeWzjjsraxwjBqi8YR2LoYFLfcozNIrD2bqcAOk2V+q1v/295yDeTTnP9 TfyYNMuGFwQjegWuOVfz1GKj11hEcBHgzfIvjlO8LY2ikBAtYqRoDenRCee7HmvBFb1A njIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686273534; x=1688865534; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=s/wneZk+WXTrjMA3fZuua9PXiv3fUuHnEiWoD80tgK8=; b=D1Op+U+gFI9vbgurzG1VKtoJNI74Cs3Mjj5ZmB6fjMx2C1ZM4Pb3/De7wnEudKU+0r pIY+6fSnzcy7BtIzVsiCPzf9MLuh3mx3MNtQRT+i9pRNZup2fi3dERUz1NdLAXtfY3oP 6AHgu0W5/yo9tVSw6L7/27FLSadUaXgRoSmiArFQMAXwi1xf7Dv/YjEIdvcv30/NiSTh 1+GeXhdr7fwwoqX/7KVASUt95MTzg29uo4Xz75SlQuoHXEDh0Oz+zD5s3igyfBvYR2LE rVXzfMhzcMn1YeFXbx+95Q4bu+l+FA2JSL0DFiDljEBv3UBlaFkB7gpiHp2hH4I0xJSu pWeg== X-Gm-Message-State: AC+VfDxhFtmUtFBeGaassy3Q5SGo+LHm+pYaodjNyc8s5zrfnPKnblEU dWCsYRCA/DfsBlSajIeIUQIGAA== X-Google-Smtp-Source: ACHHUZ5bBUa+IZ+zQxwNmCb4pbZbizWCezt9RItrFgkqOHN09PZD1rbRUEyPvItVL/IZu/e+lOkb4A== X-Received: by 2002:a25:d757:0:b0:ba7:ff37:4603 with SMTP id o84-20020a25d757000000b00ba7ff374603mr1178588ybg.45.1686273534119; Thu, 08 Jun 2023 18:18:54 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id v38-20020a25aba9000000b00b923b2935d9sm603286ybi.20.2023.06.08.18.18.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:18:53 -0700 (PDT) Date: Thu, 8 Jun 2023 18:18:49 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 10/32] mm/pagewalk: walk_pte_range() allow for pte_offset_map() In-Reply-To: Message-ID: <3eba6f0-2b-fb66-6bb6-2ee8533e221@google.com> References: MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 33E2F18000B X-Stat-Signature: 5jnmm9iduxkb6c51xpm6rfiq3zqehq5t X-Rspam-User: X-HE-Tag: 1686273535-494487 X-HE-Meta: U2FsdGVkX19p5lz95ktj7lNgX0ayDZ7Rip2Q1XJxBniyT6UAZCKTpMBcRlj7UfLAdIFpEwSc2iVClhdFeH0asZvGWTP5LcE8jdqExtqmycvL+uvGjaPiE4YCQ7xpNlKId0zWK7uZqxOrrCHxU+xidMD2AVdbTCuvYf7ewCCoKVoKMKp+bnQ5KSnGPW7nL04raj2ahxZtsuK+1sbbn7TL9mlr0XFRXaI8sN/dEOm1+6OMeSpVLUmeBO7N6/zY6fasctKc5ziopg65PeD8XBRKs9+N0tgWDtE0wqkdQQ35t1uW5lsNg/Ir85FlPEtm4r2aT8iZSLW7Ub7ElV7lYUQA0h10D9OpqPE27ZrWyLbk9OvVjd/64kxYHD9CpCEH6wupUkSNdjR8zUgMlwei5CxMOfLdLGogbKCvmGuf0nuyQ6u358p/V6TFJT+/o7hEEjuf5hpYRpF6ML+lesRE4xaPS/i/plSfb65tKEWfmAgR9xXsr+t7mJaVsrnG7vSfwHxyyv+5xooczBopkoI8W177c5Qxz9Z3O5Af9JWx8XI9EGwpNCQZ4x8YElSSzNfOEQFGJ3r6Pva2eJYK45qsCAFwTehm8RGq4qjJ0L92Yz88S5y4ElLMu4Vs0Hiw5gMBl92IJijK2LVeS2gBnn//TLxfbZcPLW9QDkiTvDnzEcNsgrOrvEdlWSehaLJkElWJ+v+vQ3jyzml59pbmMUSDXrzM90A8J3z3lPrdV5ROoLME8X1x+T43IoWJTLfSJQmPPODLU2mwshhIHjno5tK7RBsC+5hBfR7ce1nMm3MKTxwSE3nVWLEoxbOt9TS9IG+qedplLAqIAU0jTj4NUdPYFpmICJc6Hk9SdHBBV7uuw7cHWLiKGyGJc/qerKaOZvgoBt6pf4BD+BUgier9SpCfrzrYBCi9Vy7OaSEw6UgRsuce/lmH08HxwqeFKYl/5N5jOJNQFpgatldpQmo3dbLYTJK wkpIYw14 s6SsEX8ugdPvzRDMappQRqKjFTZYFQE3m4Zww/dFWYxXGZicbgBddV+3y2W6F8PCqFci33A3g+1hpVvzeHAf8RnNX7QPM/KsNGwDotRm/sMiIXJ4pEkBNen+66+/cAlZdkMscjy9ldVzufLeRTWIXllsIDbaNcRrWeMnyxrM0YrAkgR2LHerbPcWy9FTkn2L8o7pPjiTE0naQZWFvpmA3N/SlHgjkAyGOsd7YoEk25yJxmwawlNNJM7FbynvUS8Le60UKc6/14ItTMnbJh+dV4B37Wl9hM9iTW3CQ2++6qEdJsXrMcwMhlNAhOVaK4JFCVrVDN5Tdtndf4jHmTIC5ApGsO5292SGqFUwFfmsFmwrX+Pupe/qprVkLP18I+/DkkgHloPKq8XHxkCSdno0wxStHxtAOughCJmerApBPtv7keHlxMDHXK5jodQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: walk_pte_range() has a no_vma option to serve walk_page_range_novma(). I don't know of any problem, but it looks safer to check for init_mm, and use pte_offset_kernel() rather than pte_offset_map() in that case: pte_offset_map()'s pmdval validation is intended for userspace. Allow for its pte_offset_map() or pte_offset_map_lock() to fail, and retry with ACTION_AGAIN if so. Add a second check for ACTION_AGAIN in walk_pmd_range(), to catch it after return from walk_pte_range(). Remove the pmd_trans_unstable() check after split_huge_pmd() in walk_pmd_range(): walk_pte_range() now handles those cases safely (and they must fail powerpc's is_hugepd() check). Signed-off-by: Hugh Dickins --- mm/pagewalk.c | 33 +++++++++++++++++++++++---------- 1 file changed, 23 insertions(+), 10 deletions(-) diff --git a/mm/pagewalk.c b/mm/pagewalk.c index cb23f8a15c13..64437105fe0d 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -46,15 +46,27 @@ static int walk_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, spinlock_t *ptl; if (walk->no_vma) { - pte = pte_offset_map(pmd, addr); - err = walk_pte_range_inner(pte, addr, end, walk); - pte_unmap(pte); + /* + * pte_offset_map() might apply user-specific validation. + */ + if (walk->mm == &init_mm) + pte = pte_offset_kernel(pmd, addr); + else + pte = pte_offset_map(pmd, addr); + if (pte) { + err = walk_pte_range_inner(pte, addr, end, walk); + if (walk->mm != &init_mm) + pte_unmap(pte); + } } else { pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl); - err = walk_pte_range_inner(pte, addr, end, walk); - pte_unmap_unlock(pte, ptl); + if (pte) { + err = walk_pte_range_inner(pte, addr, end, walk); + pte_unmap_unlock(pte, ptl); + } } - + if (!pte) + walk->action = ACTION_AGAIN; return err; } @@ -141,11 +153,8 @@ static int walk_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, !(ops->pte_entry)) continue; - if (walk->vma) { + if (walk->vma) split_huge_pmd(walk->vma, pmd, addr); - if (pmd_trans_unstable(pmd)) - goto again; - } if (is_hugepd(__hugepd(pmd_val(*pmd)))) err = walk_hugepd_range((hugepd_t *)pmd, addr, next, walk, PMD_SHIFT); @@ -153,6 +162,10 @@ static int walk_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, err = walk_pte_range(pmd, addr, next, walk); if (err) break; + + if (walk->action == ACTION_AGAIN) + goto again; + } while (pmd++, addr = next, addr != end); return err; From patchwork Fri Jun 9 01:20:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273074 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3EFA1C7EE25 for ; Fri, 9 Jun 2023 01:20:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D16868E0002; Thu, 8 Jun 2023 21:20:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CC6ED8E0001; Thu, 8 Jun 2023 21:20:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B8F8C8E0002; Thu, 8 Jun 2023 21:20:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id A9A258E0001 for ; Thu, 8 Jun 2023 21:20:11 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 7AF5BA03B3 for ; Fri, 9 Jun 2023 01:20:11 +0000 (UTC) X-FDA: 80881453422.20.6DCB3C5 Received: from mail-yw1-f177.google.com (mail-yw1-f177.google.com [209.85.128.177]) by imf21.hostedemail.com (Postfix) with ESMTP id B113B1C0006 for ; Fri, 9 Jun 2023 01:20:09 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=sbrAsg5W; spf=pass (imf21.hostedemail.com: domain of hughd@google.com designates 209.85.128.177 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686273609; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=W3t6elq9R4Rh1GonIcnIp9HMVNr0t0fb96tM78h+6lY=; b=DGccTWVTMZZx10UV1Pz2j2KI7JdcOmW3UUHzD9aVwCHlWsdIneo62JW3OdTOhuF8MPskh/ p0/AvTU2O4KNmysnFC6GW0JArXBDkDccGWGMaOEiWpLAUeL/nSOfwQVfr08mZh44TPexfp iFxyFf1fZK6HzlCNbYWNn+B/fvucFcs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686273609; a=rsa-sha256; cv=none; b=zLyRegngnWyFkGiCtqa2Q/qMLjS4cq9qyq/UsQ3edgt/hcBYyXAteSXJGCv5JmOXut7CUE tl3b7J2SSD+ADRVUwHA73hKk0TWmrQval08uRob23sq9m906zf5Pr43wYGrzrd9Ty40Eem /05FlvL/wxKKREiziuP8ezfAurLwcUk= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=sbrAsg5W; spf=pass (imf21.hostedemail.com: domain of hughd@google.com designates 209.85.128.177 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f177.google.com with SMTP id 00721157ae682-5664b14966bso11243037b3.1 for ; Thu, 08 Jun 2023 18:20:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686273609; x=1688865609; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=W3t6elq9R4Rh1GonIcnIp9HMVNr0t0fb96tM78h+6lY=; b=sbrAsg5WUYRFtaZIknvbQiG6/TYzlsI+0wgQgiUfJThncdOkuzQD0khSKs1V9z07r2 Tv51pW/PAoavvTS97xjUmu8LWWg94YJvp9vB5pfhhczalF5/DDl+Uv7HP4vGiWYqNo4X NvPImQC1dON//ZIIaJuY80thsOcoTmEarD24++KqimOdduTFWASKv9w7+mCL83HP6Bjj lCLJbqulD5C8g370KMX/kus7X70GOqLQ+qmTon2m/00V/c4ahLqm0Dsyh/zh31jb7WvE PrxBXcgSpFRmWGUOtPIfuBbXHPk/T7GjZ0IZtugAqqqNtfJYQbkhOVtUktXosevWE1uY kB2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686273609; x=1688865609; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=W3t6elq9R4Rh1GonIcnIp9HMVNr0t0fb96tM78h+6lY=; b=d0ydeb6jb17LrUQUKfc8tIkr/f+0CbtSRvkfiyJTxIGhwHUM8Rx6igwYOUZ3KygEKa B3xhvnaZha8ljiqio+bWtDkN0Kyxt49w6KOr9LKwtUNMzlUP4F9HY3A6UucVE+T/dvXb zaPgUc7dmHG1S9uV4P4MFpmtO8T2YqfKfhfvhMRCY/zCDNXz9XXeNtVxIFoKhi+yQ1wx +XrxEaMTtxE+odEXZlfVzoBpn0LuS/TRAIG10opnqfBp/R6XCXc52m3JjGYqO9s0N0Bn GYbA7X4vn+Ed16B7DwQX30tn64a0AHtD03Ypmkne5JRGLc66Vk5qqqlrGqKF7dbfzEXg tLqg== X-Gm-Message-State: AC+VfDwUM9EOyvw7QCu7/IM4Yggz/5aH85mWVc82na96OmaXe1SDku+S T2c1TXjVZcnZP4qClB8c8lLN0Q== X-Google-Smtp-Source: ACHHUZ6n8W5YFW3Tw+jdRjglpr2r72K5tunXYPvH/mLAwhgJM3++HeY9qf1iqDSmIO6b6XIA/ifN5Q== X-Received: by 2002:a0d:dd01:0:b0:568:b0f6:ce8a with SMTP id g1-20020a0ddd01000000b00568b0f6ce8amr1205033ywe.24.1686273608570; Thu, 08 Jun 2023 18:20:08 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id j77-20020a819250000000b00565862c5e90sm289860ywg.83.2023.06.08.18.20.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:20:07 -0700 (PDT) Date: Thu, 8 Jun 2023 18:20:04 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 11/32] mm/vmwgfx: simplify pmd & pud mapping dirty helpers In-Reply-To: Message-ID: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: B113B1C0006 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: u49q5kg356i1zg5ca5ih7j17fi7u9u6g X-HE-Tag: 1686273609-832115 X-HE-Meta: U2FsdGVkX1/ovSjY2usCjNeKbqbBEDye31Qq+O7vfou0jm1m0jzv/a6jyW1WuOjpe5P3Wg4Rel++DZ+tRIIccaLmWriqLHlMCYX2tfumyr0ETgnK2QiIVvTaWjSZvIA+BRFDJIs4yReYrcy+oeVLAJrieRbhpZP9py5bAAXy2Nd1gtMGAc4sJVCN0JyLHXiQN2EV/BytLjvEsjhD+UFAuaKpDuyxx7iwR9n2k/WybYAB49Bx99ya5RfM1EIAR0Gtbck1VbcRAIng/DJ8XR5XYTzezC4SroLnYZARVAI5Jp6Qv27ryxryMsSSAxxzBpEa3AePeuMRLvPVx5ip3QS98ePPfbKMyK+vukW8Lfbt2CP6ljH+bcig1TZprx83TravxXBUAOK2PwqRO9dp1E98bi3OFvqlBV02JHAZxd5uE/R9sKZPeq3ad6nmhAcA5/vDLlMg4oUeZ1XxDrvGVOitOzKAZcEoVz3qd5qUlMDIpKbDnZ9p/FMYrl2inESVLCp+W8k1VLmTGUzBatEMpBqqMHAFD/VDvFFzOyRMLWpl4Uxhohmvam3FNkh81obYVDE39pibFjpn25vei+tcYQowlVh5d7IDje1VL5B8/wxcM8r1iypTegru0uiqmYYPM6oX0HNIhssWXH9AMZ7e4/gGoNUr88FV/TlmWaO8vtOBWBc8YpFhekbinS9hlmKIMOyDKEBRNRjhOgWNQbsQv3H2JB6UlMjAqI/LuS8QB0ALRc+JC1IN0tzUg2GFsgGEBfkayfNhSuIe9UGvJSJHQjJEkxqHIoUgNDW3KOdeABj2a2z4t9dRzqhw582BB6Wa+UHWuUGyLo3QYbDeLhl9qLIsbOEwgPTGRXW9rkGB52doGSWFqD6Ni6gnSB5X9+ltEz4EftkxQUBNpDJcurgpi9HkpUZR/n9bcFHVmNA28cksH6+U5rgUEDSFlCNOiBOlMDlQdawF0VsqJxhmY/4F+SP FUGIZsEA A1pkv4xaqzbSqjRzdDqnAInkt9fe4L1ZKhdwuaXXN7Pu/2skgNdQmwr8K5ZaccblEHaPBAwAGz3X5TSr+YKYfZH1GSkNJQdsxjp5RXXFNpFmze1GJ5gyjtT58fgcjwS8jPNX5kKZIM438gqwQOKqsrV4AEmI6oJqLyrkfLjjeHOuYRjCeeoTp3Bu29seroeadmOR0aQ3aEvjXnetYQTzOuqM5LfLbMEc/yuIvX1L3wsyovZ6Hf6eZ4Y+UAHb215d/11WM/1tExkMb9srK5CmIu++8U1+6yIuk74jrMebszYEKOVu/OJEXhzx3gKuMIDZXnGnV875hLHpTpFq0ShDfkbi1YotM1R90YikXB5LJm6aveG1n8kdZcSotoy73DuufMBP1 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: wp_clean_pmd_entry() need not check pmd_trans_unstable() or pmd_none(), wp_clean_pud_entry() need not check pud_trans_unstable() or pud_none(): it's just the ACTION_CONTINUE when trans_huge or devmap that's needed to prevent splitting, and we're hoping to remove pmd_trans_unstable(). Is that PUD #ifdef necessary? Maybe some configs are missing a stub. Signed-off-by: Hugh Dickins --- mm/mapping_dirty_helpers.c | 34 +++++++++------------------------- 1 file changed, 9 insertions(+), 25 deletions(-) diff --git a/mm/mapping_dirty_helpers.c b/mm/mapping_dirty_helpers.c index e1eb33f49059..87b4beeda4fa 100644 --- a/mm/mapping_dirty_helpers.c +++ b/mm/mapping_dirty_helpers.c @@ -128,19 +128,11 @@ static int wp_clean_pmd_entry(pmd_t *pmd, unsigned long addr, unsigned long end, { pmd_t pmdval = pmdp_get_lockless(pmd); - if (!pmd_trans_unstable(&pmdval)) - return 0; - - if (pmd_none(pmdval)) { - walk->action = ACTION_AGAIN; - return 0; - } - - /* Huge pmd, present or migrated */ - walk->action = ACTION_CONTINUE; - if (pmd_trans_huge(pmdval) || pmd_devmap(pmdval)) + /* Do not split a huge pmd, present or migrated */ + if (pmd_trans_huge(pmdval) || pmd_devmap(pmdval)) { WARN_ON(pmd_write(pmdval) || pmd_dirty(pmdval)); - + walk->action = ACTION_CONTINUE; + } return 0; } @@ -156,23 +148,15 @@ static int wp_clean_pmd_entry(pmd_t *pmd, unsigned long addr, unsigned long end, static int wp_clean_pud_entry(pud_t *pud, unsigned long addr, unsigned long end, struct mm_walk *walk) { +#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD pud_t pudval = READ_ONCE(*pud); - if (!pud_trans_unstable(&pudval)) - return 0; - - if (pud_none(pudval)) { - walk->action = ACTION_AGAIN; - return 0; - } - -#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD - /* Huge pud */ - walk->action = ACTION_CONTINUE; - if (pud_trans_huge(pudval) || pud_devmap(pudval)) + /* Do not split a huge pud */ + if (pud_trans_huge(pudval) || pud_devmap(pudval)) { WARN_ON(pud_write(pudval) || pud_dirty(pudval)); + walk->action = ACTION_CONTINUE; + } #endif - return 0; } From patchwork Fri Jun 9 01:21:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273075 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8499AC7EE25 for ; Fri, 9 Jun 2023 01:21:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 094398E0002; Thu, 8 Jun 2023 21:21:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 06A768E0001; Thu, 8 Jun 2023 21:21:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E9B1A8E0002; Thu, 8 Jun 2023 21:21:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id DC1698E0001 for ; Thu, 8 Jun 2023 21:21:48 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id AC982AF050 for ; Fri, 9 Jun 2023 01:21:48 +0000 (UTC) X-FDA: 80881457496.08.25020F3 Received: from mail-yw1-f176.google.com (mail-yw1-f176.google.com [209.85.128.176]) by imf04.hostedemail.com (Postfix) with ESMTP id C4CCF40010 for ; Fri, 9 Jun 2023 01:21:46 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=DCQmA7jy; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of hughd@google.com designates 209.85.128.176 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686273706; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ngswcYXh32wJZMu4ey5160LWyE599vrHnWvZmx6kKak=; b=l7WzvKygXSlCfO7FO9Yay1wfp9IeuCn74TWLoQEwlI68yt10NPDWF7/W5dQu8kOc0XO2Xx 7Ck4kngMZSoL66zRoLCmOTXrC3lHD/0G795a326OTKDrkApWQGymGR81dbp5PiR4r5dpNN UidNmhT54cY0JSoSlU20QG2L6K9XBWQ= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=DCQmA7jy; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of hughd@google.com designates 209.85.128.176 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686273706; a=rsa-sha256; cv=none; b=Uf5LyP41MrWDDpntGAvqSefAp/jeYNTQwDuOlEo2bkQCuo3fub3ZGjaDWCEPKXdHFSgear aEaV0j6tT3n3yiDzQI6p6xySbga6W3YPeIuVpdKWm3h5aDw6ssSdHGnPsPUrtsZ8R73DDb ejNjK4bNTEjjCqZWdMITWu4/w7rfk0s= Received: by mail-yw1-f176.google.com with SMTP id 00721157ae682-5659d85876dso10757407b3.2 for ; Thu, 08 Jun 2023 18:21:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686273706; x=1688865706; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=ngswcYXh32wJZMu4ey5160LWyE599vrHnWvZmx6kKak=; b=DCQmA7jyE39MLlNzgNj5Fnvt6VxEVhxfR43nyr9O34TsE/NR89GCRNiD4ZoZdR4Qh/ VVlLGnas01pgDThKof5oMLzzf1GcWd8TwEjNBhNMTY02KeG47WG7BA2R3pdxHonu/WZa V3CvdasgvhS1HvjB0i2UUPMQsDXJR6NqotUE643j7wgPp8AFNy/qFFWZUuLH+fdomurR thKNoJK95KQaiNcwFeztAb+d5ZaqpXtxhvroVgVoxsGEe35huAPILPbeWC7G/RPecM4M XWHbDzBCI7PfoB33hTMYlIVyb/ngBEvTOCi3mr/ilks4HMOz+anwKzwFIqfyaYnbXfBU DGJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686273706; x=1688865706; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ngswcYXh32wJZMu4ey5160LWyE599vrHnWvZmx6kKak=; b=N6h7qAJ+WrH4wEbfX3C0JNUu331JiWJODXRRlRSIsnFyoKE3ajjTklgvfXPjFIM5fo 4rVCElpcknYBst7SWrq3itTBJ2/JcW3sBwE8OI9+o93u1048geq4/ClveTOmZORZOJ0C Gmv1yc5JXpFJQTpKCFUbggE4dJOH3zu4LXG/G3SQ5F8FvXLLwCCKsW45PZoZSNBYhQeW WZ3u1v+a23CtXAdfcgOpDyUM2XAYNKz7Tp7Q26jl7gTev4nzqE3neMT4j8VBRQDYOLy4 Ih7zxO1+S0cicStz0IwfVDUJ/crmFpdor+J6JhATx1PU2vJhILW+y/gYqVAZmWPgXIn5 DmLw== X-Gm-Message-State: AC+VfDyUWm28sxBsgvXYn2hHLRhxZViTomCBmxxkgrh8FYJ72GMxfAJc ac/WlwogdchE0iFUTKEGcBa9Ug== X-Google-Smtp-Source: ACHHUZ6Ffc3fQZoL4RMB14OmxVVM/4jw5aXSDM9hxxoBcUG6hRO5j/bd0jr5kM6GfmtOtWXvD7TQfA== X-Received: by 2002:a25:f501:0:b0:bb3:9255:33e9 with SMTP id a1-20020a25f501000000b00bb3925533e9mr1145250ybe.53.1686273705789; Thu, 08 Jun 2023 18:21:45 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id t12-20020a5b0dcc000000b00bb138b444dcsm586743ybr.36.2023.06.08.18.21.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:21:44 -0700 (PDT) Date: Thu, 8 Jun 2023 18:21:41 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 12/32] mm/vmalloc: vmalloc_to_page() use pte_offset_kernel() In-Reply-To: Message-ID: <696386a-84f8-b33c-82e5-f865ed6eb39@google.com> References: MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: C4CCF40010 X-Stat-Signature: bgw65wyts14z8w9ifay5irnzewp74r1g X-Rspam-User: X-HE-Tag: 1686273706-895999 X-HE-Meta: U2FsdGVkX1+aRJ3iyfPfDrdBtXdeE9EZ/huN0s54TfN3KVTBmPzZjDGswOUMND7FUyHEH5XUnB4MINr42MgnR9T7SgpfkZmgPZ7v7mDzRlua8TMQFfBH8VS4DTOTeBT0hQd7kHPQ/CHgpWFdCi4FHvZGMIf3ZlZWPO/QbtNN1sC5xwsnes2zhoNwiUF7uqSgmiqrvSSq2NkPZRhlk4SmhI0g4zjIWq3YYBeI/WGArDLekuAdHGn68a1OmkNW8+Ce4ZYMeteMkvnMNSQOmfInhzez6fnh2B31+uL9qB1hbXh4mEtsiCdAHiCs6oRVdNVUP/413nGQJqFcyOks2MVMrPMA0GN7j7akShdAfzDY34bA4YfEHGRt3sgSbT0bWywjnf2B9x5W9upR7cjWO7dxC/Br5B2WZQBPF8LOtRRuK+CmVhQ+/k5gVkkdukUsPNuu85EKe0ssaxOGOpgTz+7dAXxvgZEcKjgsL1ZEvcvY9zpmGtQTLWCRil6iT3zQrsRsE/UXJdXa+FaOVZlXw7RCqhxbl2vLiRVP4euyRwCcC6Jw9OQSwXs3BN0AcUMaqDo3Vd8HLSqZlIf07Mi+gFD3n61tWlRwwP5SamSsUaaM5BNoEHBNkrF4/YU/+GsVwI6a4T+lt/moN0J586v7k6qBSelfbeL4rApXJ6Aa2Jn8hU+cIM+ccqXC8NC5xCwyGLXvq/+nUwriL/wwxumklcr3HKeoLXDAEjFK04qeraiVDgF+4PkDNIVtejvjt7BkiF615X7ZD45icDfN9UUe3FH/znQey354DjahYLJl6VISXutW07X6ZdUxLIFvOtNLtSuIIja1bDKzu4c3GAXzveFgaOL1q789pTPNfaAsy2ZpAiD60LhThyMYYU/0ZKRLmlY22DTJqvh4+5fg2OrNvdmlyRlDlN1oJE5Qm+a4B2Shh8YxwDgDUq5j/Rgwui6hLNByndyFDlzeD7affx9QD4g IDA6KiJJ EYQqcecQXrHCJ4D+Q2XHEP/Wtjn/NB9DUkEoitJNFntOZWJWUjkNMRM3iAjY9sWG8IZM6Yo+p9mPhCZoFJMXmDyrYXh/MgsOtMaym5goH0ZqNcknpZjN7kd8OlBxYHPV6v2UTVMegHP/RsaK7HoO282EPIVsDG53O9+OXKXlhJiR0N1nO0ah13OhNINWp/up5YD6KrJgbVmNKp4ukMweV7HblViXGix/e6+Fp47vuglt9p8IbglK14bMUHZyJZzE9AjqYvWkOWwK/ujpBeONvdmzZLWFXHn/naqfWoEUgQreBqtfaHo2106CPZZNEjiVzNG6/vbSVCaEEX1em8e1zxkRC6xTkBnbqkBTWW9/AKmKDF2BorAz7yXruKIhfTpF9tQbcFuFYkD7xuTWNJOZd3pTqFHw4f98K4bJYha6w7r8WnRcW0/0gKaCAJ9CSpLAXViMkvyHZELzus2I= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: vmalloc_to_page() was using pte_offset_map() (followed by pte_unmap()), but it's intended for userspace page tables: prefer pte_offset_kernel(). Signed-off-by: Hugh Dickins Reviewed-by: Lorenzo Stoakes --- mm/vmalloc.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 9683573f1225..741722d247d5 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -703,11 +703,10 @@ struct page *vmalloc_to_page(const void *vmalloc_addr) if (WARN_ON_ONCE(pmd_bad(*pmd))) return NULL; - ptep = pte_offset_map(pmd, addr); + ptep = pte_offset_kernel(pmd, addr); pte = *ptep; if (pte_present(pte)) page = pte_page(pte); - pte_unmap(ptep); return page; } From patchwork Fri Jun 9 01:23:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273076 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B40BC7EE25 for ; Fri, 9 Jun 2023 01:23:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E277F8E0002; Thu, 8 Jun 2023 21:23:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DD6D38E0001; Thu, 8 Jun 2023 21:23:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C9FB88E0002; Thu, 8 Jun 2023 21:23:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id BC55E8E0001 for ; Thu, 8 Jun 2023 21:23:26 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 93BD4C0329 for ; Fri, 9 Jun 2023 01:23:26 +0000 (UTC) X-FDA: 80881461612.22.BFC3234 Received: from mail-yw1-f169.google.com (mail-yw1-f169.google.com [209.85.128.169]) by imf28.hostedemail.com (Postfix) with ESMTP id C1474C0009 for ; Fri, 9 Jun 2023 01:23:24 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=PhwpDxcj; spf=pass (imf28.hostedemail.com: domain of hughd@google.com designates 209.85.128.169 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686273804; a=rsa-sha256; cv=none; b=MLkNXGSWSBSREvOW6Vztx8Umz5DmvYDw39pv1sAgWQ3ZsVLthPunXCEI7ylaDkxVPWdaY1 FSZayv5zEdU9I2q3fGiUgewTezXvZCJEyxPVOiI4dtyj7dkg/Gv5zhdqzE7gwvBs9IhMp7 OxSTkW9yxMp/uohsgH5Y4Xnkup+hpPc= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=PhwpDxcj; spf=pass (imf28.hostedemail.com: domain of hughd@google.com designates 209.85.128.169 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686273804; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3KTDoQQSpcjB9NQiyOQFUXOX88v07BrrKabYvpMIx6w=; b=IivBxkyRlenqAEwhwhwBpcqG75sPh1ouaRny82bbueyd3U5gbtZ+s1Y4vvw0SAC7T7sbAo Vc3/bhTPbnXqHVmawes7cuScxg6OVyFeMFCC2txm8hnQHw6pXVIWhXlNESgtORGMNz0/aF MOb5KIJDq8awkAmXSZF34Psr/CJMYnY= Received: by mail-yw1-f169.google.com with SMTP id 00721157ae682-565cd2fc9acso11719677b3.0 for ; Thu, 08 Jun 2023 18:23:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686273804; x=1688865804; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=3KTDoQQSpcjB9NQiyOQFUXOX88v07BrrKabYvpMIx6w=; b=PhwpDxcjLXiR7MGVaWKfABLE3dmRB3qka2k6U1l+muDGzWLhomU4qiQ57KmT6VH8qY ACB3DayhC4faseiipd0QCNLUtNtuFH3XTvyWMyC+FvmNiYj2hTBmc8pdfyd4XFmOfbWU tZooxmyCXH+STIuVMaRhFjnJtSpEto0Mxch6xRdf+bo4VO2G6a/c8gheC2mOEmzSx5zk ccYQOeUWSm1v0s9lMRW9nRmwt5CbXGDSvPCaqv/QLtUPzWt0If7EQ4o8TmUAMcKVROe3 I5HEpFf63MgdvN7DdkYJ8qdg2z230og4tgBl+c7UnWgkZm/cRQnWkzi++q5rmkDVXcaG 0vug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686273804; x=1688865804; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=3KTDoQQSpcjB9NQiyOQFUXOX88v07BrrKabYvpMIx6w=; b=YPVBFHf3ywBIsprmSpTbM2Im41Wbos5Hrl+Hb/joLksfRTL5faKEWQRMJGqWVLsl7h jNsxMRukQ9E9bj6vYxwLOmrhHCVtIGnurxVjedlUzmbgn2noLAss0ajjmYIU1jv2cLSW b13HfyqIcOPCS36hTmGcxrU1xw2khIqBxPYLkvJ9vg1w+r/UrwB1/aQP89zMqHpENlw+ vuXq0nmduoJK7cyi86r2iVIEvhyjGai8126OctU69Jdo2rlFxxMWRYBmZNgDcykFmAO3 DGvTxdystqnfXxJic/QIV1cipWpWapxHkqBDGXcgU8hdLVnjPLydJ+8gxrdeNoCXnQ8u 0jYA== X-Gm-Message-State: AC+VfDwyKGTXFKO8J3lo5Y7sAO1lUNmd/MsN2OvqgYtw3O2IiBikYzFg cA9vVQ0635szPWbpnWmTry/swQ== X-Google-Smtp-Source: ACHHUZ5Qxg9qOQctVm9TDBIrvRBwWLkj4+fag3kdEXz2orEpq2AOsjXF+B86JD9LcmOUKUNihoCjxw== X-Received: by 2002:a81:6743:0:b0:568:f9f0:b057 with SMTP id b64-20020a816743000000b00568f9f0b057mr1198700ywc.26.1686273803806; Thu, 08 Jun 2023 18:23:23 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id i11-20020a0ddf0b000000b00568ab5dd873sm288724ywe.65.2023.06.08.18.23.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:23:23 -0700 (PDT) Date: Thu, 8 Jun 2023 18:23:19 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 13/32] mm/hmm: retry if pte_offset_map() fails In-Reply-To: Message-ID: References: MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: C1474C0009 X-Stat-Signature: pmgad7hoq4wbcez4ydenxik164xa1jua X-Rspam-User: X-HE-Tag: 1686273804-326053 X-HE-Meta: U2FsdGVkX19AQ5WNFe1PSDOkSKGNAVeIjH9Ssoo13Is64gQJ7jNMqMCKNWWc+NgADf73ZOD5syCXB168sam21ruqztocxc4+nf7ZPBSNjDekCibC4NexHYFpQHKFhEk8styCjev5SAM2Q6b6Xua3BJeJDSXyqdVmQxtNJTgnPJDRexUkJN12Ugtj+/GNGVult/U6hDzSybldeKN8Uwo55+Wrd53Qld+C1D6syc8UHLvvUCQ4rmq7IDV+XWBL4IKP4PBDrs39NTLW+p3oCm5zNG0nC4qGeShapfw8Jq2YEUXNOxfhNVMbPLzesmCvlMl9eJRhGBC1WOqFi/OhHh77Nx+tA9yO5mL28tm+MNa2AlAbvEOm4gIXjTTKnSHlpAiNKF6/RhzqOUejJXGPCeNNloUHVp5sQYPx8E1FuMEs7jSaShD9zSI3/+YDpmNMPWfSinCl7pbCX2xpeoK8A5RTWqk4hcnOAgqq95DnHn9C2izmv5HhSQ7/nykja/I6R1xFY+AlWjU4n1QmyKj2hp9uzS0jSpVS8SWQz/96UXnkc5Gfe6MzKg3nE16dXi743KJjyO9mrFqzBqfeK3P+REOY7pG2j7iZ8v6OHC/Au8ZdL4tVOkV51pXOlXxQF+j+Lsbj1Aw4pFvN5haXAuulnAKIjKH50lf7vEb5NulPgkttinyPj+UL8mixIwBx3brhTfRqw773GbN+8lWy96SNV+PM8PU+j73rOk6YA1R8LXpPJNq8A6kCBASUHhJ0tntlcXY2TkHxsB7ecB2MLVdCxIFPJKS1QL6kEDlSe4YEb6UB/LfoGgMJYev3lmk00BthfcDnL4jc9UFoT8pa8NS3Sj66J+3La1DFq9In4RfTb6wqOwumsIBRZuOgNPbbX9SWLUG2u8HYgfLb8OHfPLhXhZxm0PmAJ8T/2jpW8vCfVgA0w4vLfHmUEdm2AE1oT1LK4u/Ulr+ISb2zzzZN/ztSxCY tLKslW7I nbZaXIgY/rA76JW9+0WJBE4cpUT8r1xToxM0K/8sk6qtGEfDV00uXf5/EqpTWYe1dOrQ6fN2bIYRjIUEgvl+cFUY1kfnttSmzeUnzqRLmgCSghVB0fsyUlX3GYbkQMV9Bk49VlzJJnzAs8WlXPly84AVxMMYVnq8Qp/0ZmR8XZyJvsiQsXpIedlytwbDpLnzBUSpX5BZ14tWdagvbddP0ej+TJVx9QVX6f8QfL8BXe+4Kilzdkgcl2QTQPqU7mdEJzF103aXydU5pl/Z487prya0/9vNjvCgWi6rTNjHgWtI3NKE0ReZTpo4Kqf3HY+QTJPE6m8Kmmhhw5SPFTf72QjHpbiPSBA8A8kd8dhiZBmKxZhSmRCTEgJiFiFXM3+B9CRTfpLbGIjlWflC+U4+llmB6D8WPm6aWYqoLdShDO2dO2NwGgySQrFH/5XjzbZwhDO2v5QH0SFs/O84= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: hmm_vma_walk_pmd() is called through mm_walk, but already has a goto again loop of its own, so take part in that if pte_offset_map() fails. Signed-off-by: Hugh Dickins Reviewed-by: Alistair Popple --- mm/hmm.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/hmm.c b/mm/hmm.c index e23043345615..b1a9159d7c92 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -381,6 +381,8 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, } ptep = pte_offset_map(pmdp, addr); + if (!ptep) + goto again; for (; addr < end; addr += PAGE_SIZE, ptep++, hmm_pfns++) { int r; From patchwork Fri Jun 9 01:24:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273078 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B8B4C7EE25 for ; Fri, 9 Jun 2023 01:24:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1F1458E0002; Thu, 8 Jun 2023 21:24:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1A18F8E0001; Thu, 8 Jun 2023 21:24:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 090858E0002; Thu, 8 Jun 2023 21:24:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id EFF868E0001 for ; Thu, 8 Jun 2023 21:24:45 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id D05241A02F5 for ; Fri, 9 Jun 2023 01:24:45 +0000 (UTC) X-FDA: 80881464930.21.BD255BF Received: from mail-yb1-f182.google.com (mail-yb1-f182.google.com [209.85.219.182]) by imf23.hostedemail.com (Postfix) with ESMTP id 1183D140005 for ; Fri, 9 Jun 2023 01:24:43 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=En3MWCFj; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of hughd@google.com designates 209.85.219.182 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686273884; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wNia0t/q7bIuS9YTvnKusLroNIjYrblkNZ2RtXpvdKE=; b=WGVeP/iSQs1HiyUMs6xOEGx9CeC4t5qTz5Yb5N7hUjNuvSGZsJI6/HW2z0cJBADvHc0yba 1NRCaYQw4awQJtaRqKXxM0jTPwfrI6SB/8LXTCyC1XKRLXyEF4OkCLz/uZAzEAx/6oJiBp dTbg9tqL94r+rSITzOzFEBcXop/ny2M= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=En3MWCFj; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of hughd@google.com designates 209.85.219.182 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686273884; a=rsa-sha256; cv=none; b=rpu+19WCANLEBp/WpVn49YlpcXycQ4/8yrHMgMgv37nGyf07jD+9SjY0Wif41wUKDu0UNu xoiJdx4V+75nbSnP8cnRkVApCO4NWuJQ5sxZaFX4FORF3uHXGBrygSMs4A0diTuIxFdDHy RQ2EEd72IeXPddnGGQtxT5Y7ZkIX+q0= Received: by mail-yb1-f182.google.com with SMTP id 3f1490d57ef6-bad010e1e50so1319049276.1 for ; Thu, 08 Jun 2023 18:24:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686273883; x=1688865883; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=wNia0t/q7bIuS9YTvnKusLroNIjYrblkNZ2RtXpvdKE=; b=En3MWCFjZBPbHoaHE9aTCrwnidrPgxHv9KrEbLe/hJSNlvG00NQArOmMAEVh1cdPK1 c0avwttgIxMg1/M4xoSdtZvgHC6q+oGNCMIaib2sYWv0Fx3ipPlY1RUsAjzGUeuRCY7q f4C9d8VWmMay/sRpQantLYEl4VbdRT+IrwRgP0s1LNmuiH920qhuNY3jPTG46R78i5+k 9bDaQ6fLa6fnzNxc3zf976MlEqNb+2p/4gql5IeQqWDTKZblvLqIpzvT31fUaUlXU1zw KT/OmxBpL9FU5MCWQGoReuGTHeYWIqJH3GhVd3emhvyvLQrP6bSWfVqdEbmUf+1ykpPu 0Uaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686273883; x=1688865883; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wNia0t/q7bIuS9YTvnKusLroNIjYrblkNZ2RtXpvdKE=; b=lwjlju13qztAwBUY3plrgQmbg5Dn9LWS7+wrEUG8R/SU6vcsTLvYDMaXiTPU671s7V Zvp+4bgZxxCFdYBZWRE9IB2CMXDG7w9QkVzAfYeaOuEKYd3nFMljqA34Uzdw7NUS5mw7 2O0UVhqLrFX0mmCITzrgnZsregoPPfFCnYyI3T6HlRe7ODvDueSVyMgQTd/nO/dDeZGD V7bsjySU+n0X5HBgLLtRK3u89OborBW8/U1j2paU7VIBVLOVimgeY1xVv9cWBFnf9KBs 3gprkUlaYZLiNAuIJ5bGUiSOsa6wh/OjIve3sy2R5xHuS+FSXLWrK83hjBB7nsGKCHn7 Hr0A== X-Gm-Message-State: AC+VfDxVqtjm4o5lTGiEsIYSXMeBqfkU3q1IdlG9hlf+KFANjP6MdbwC QC4iofTnUMMjasIQ92wtmYA53g== X-Google-Smtp-Source: ACHHUZ7fnPyD1AB15VbBenw69LG3/w2FA1xyBpqbSoDD115cddrIVPxKnnpMPhIjJpC81nscf3jcJA== X-Received: by 2002:a25:8d83:0:b0:ba1:6bad:41a4 with SMTP id o3-20020a258d83000000b00ba16bad41a4mr1193186ybl.14.1686273882963; Thu, 08 Jun 2023 18:24:42 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id ch22-20020a0569020b1600b00bac1087b44esm587924ybb.35.2023.06.08.18.24.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:24:42 -0700 (PDT) Date: Thu, 8 Jun 2023 18:24:38 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 14/32] mm/userfaultfd: retry if pte_offset_map() fails In-Reply-To: Message-ID: <54423f-3dff-fd8d-614a-632727cc4cfb@google.com> References: MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 1183D140005 X-Stat-Signature: f65a9jjihgzymeksimocxritzqybsosn X-Rspam-User: X-HE-Tag: 1686273883-307421 X-HE-Meta: U2FsdGVkX1/2VWBCexX0nW/QEpGmuEnTdoAEYN+z6t49KlekqFYFKEQgFRrnOUfPkokTyNsYVsDaaGa/J6cg5LgpdrIuDgCIwK3Mb3I01fXSVGTvXD8wtp4TiQdJM5FMN74IpG/IlKys6J2/h0VWl6z0rI00zocripmLJomPgvBlFRImUErscE4dXWUhFICRe6E8gDnFmD0+fTMdX0W8HV+w3tJztRbqVrITE1/4Cwz5KgUBSNdOKz8HjIIsADSKXmJCUJdjuNdnQcqTwIRGDXv6WD7yLo/IPXSrnirhA/wk9+0DMoEAJ/1Om4vL6HZV02ZoBtl8rDuR/wwCSpcLzaiqpCGxNxak+iBvbPydSWMCxYND/aSuvua66m8LeNhPVYBIyCOdGTrgv192lkN2y6h6Yf4sbANqmgbsCB8pUwRDIUqGUCxWMbRbVQD5bVqRoXU3so6BJZXEjoteA+FoSnAbqw+U795EBIdJK10qb8mJgtem5xnSu0zXFqBi/71ZAPydYq/+8owE4yV1B4+1/PNSy9sz2+++AREUwg2Uu/ThwNVHy2FnRp+e975Cjy3wt/jfr07oZjBEJMGCqlI2FlwCC7/mfUC831XmN+UBxu3rAIZzwYpxOYqJ2ryorBe9Q/4xDnhstsB34XnBs9iboGERIT2Ws+PBYnLxEKIeicN2dKxWPsmp0ilopRKEmjxXIZgXsfaGqF09iBwEmK1G3W2sTCxn3CikUw2atY6XHplfH5Wb48KZSDlBum/TZYJ4MhMCbUEoadNeNIzNrcU8R0yHvsSwUJFwS1lcfapFtZqmwNowqJpe69QJ5rYzXJ70HLM13aV6Ou59F16efzqTH7AtcdgkQs2cU7we2tX2eUQ2dYjKHqHrtKqGjci2sz+R1ojlwERo3kdyQcrQ9qhlY0wz9QhohzQ0atMHZv0SKbwEPwV0p4lYaTl6zF9fSB8lkN04ddRuR8i8oUkDJkl Xwt1hgkH vtmqkmArk/U4KWRai2UTRLXMaXRr/+5YoMF+frUE05iPRiHfhuvfmoMsrJTVeGU7I6/QnAFKtSEmytdIOt+ZPbL4aGBg8kQo47HIBES5E/rM3M7pYY7LG4J/jP/MO5pQt88O/W1xyJMCkUFiMM/TjlOuZUJf80L8+iIflF/v4lUfZ7P/ktYICdzD7xa9PK2B22SUBIGpg4QAnd7VY8j+cGWXqWuj23RlPoo2QLQ6OE6ZS7I0jQ47S/ohKaDHv+v9iq8xApSLUBIFzwYj59QsAtf22oO4FMXfadk+kRTust1jn9BQV0JnE0Vvcq8l7EDUAELTwFFeok7tCeLjJ1HFNULhOtXBW8woBP4dId6EY9GdAUyrab1h1VjMAwOKjZpGydMQyZSqoQgz9enpTxjWXEapJZaUDNAWWMaNm8XRhvjwNqhr8YDOpHpb+tLdYvTupab6h9PBJCOFpBnc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Instead of worrying whether the pmd is stable, userfaultfd_must_wait() call pte_offset_map() as before, but go back to try again if that fails. Risk of endless loop? It already broke out if pmd_none(), !pmd_present() or pmd_trans_huge(), and pte_offset_map() would have cleared pmd_bad(): which leaves pmd_devmap(). Presumably pmd_devmap() is inappropriate in a vma subject to userfaultfd (it would have been mistreated before), but add a check just to avoid all possibility of endless loop there. Signed-off-by: Hugh Dickins Acked-by: Peter Xu --- fs/userfaultfd.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index f7a0817b1ec0..ca83423f8d54 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -349,12 +349,13 @@ static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx, if (!pud_present(*pud)) goto out; pmd = pmd_offset(pud, address); +again: _pmd = pmdp_get_lockless(pmd); if (pmd_none(_pmd)) goto out; ret = false; - if (!pmd_present(_pmd)) + if (!pmd_present(_pmd) || pmd_devmap(_pmd)) goto out; if (pmd_trans_huge(_pmd)) { @@ -363,11 +364,11 @@ static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx, goto out; } - /* - * the pmd is stable (as in !pmd_trans_unstable) so we can re-read it - * and use the standard pte_offset_map() instead of parsing _pmd. - */ pte = pte_offset_map(pmd, address); + if (!pte) { + ret = true; + goto again; + } /* * Lockless access: we're in a wait_event so it's ok if it * changes under us. PTE markers should be handled the same as none From patchwork Fri Jun 9 01:26:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273079 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18A75C7EE29 for ; Fri, 9 Jun 2023 01:26:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A7CCF8E0002; Thu, 8 Jun 2023 21:26:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A2C938E0001; Thu, 8 Jun 2023 21:26:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 91B868E0002; Thu, 8 Jun 2023 21:26:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 825B28E0001 for ; Thu, 8 Jun 2023 21:26:12 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 50FE4AEEF5 for ; Fri, 9 Jun 2023 01:26:12 +0000 (UTC) X-FDA: 80881468584.04.7C8A165 Received: from mail-yw1-f175.google.com (mail-yw1-f175.google.com [209.85.128.175]) by imf22.hostedemail.com (Postfix) with ESMTP id 719A8C0020 for ; Fri, 9 Jun 2023 01:26:09 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=d+dKJf7M; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of hughd@google.com designates 209.85.128.175 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686273969; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PmbbTemShg7dGpCJymh8ioCm9PQPk/gLwHXF7mQ6jbw=; b=w2JdmcKs84VlIRZJVtNXsca/Qzu5C7mokF5/6XVTt/HlhPC/Z2Uw7qdB2tS1qLgISS2qbj cdmS4yma6ZVqT7tGYLip5lopPf7Vew4TXkwKg91a1LHMWaVj02NpwJehhe+uCgZ2WfpNYV Kwon7BpztR4nXatDgNTAHYhGhRY74bo= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=d+dKJf7M; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of hughd@google.com designates 209.85.128.175 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686273969; a=rsa-sha256; cv=none; b=JB0CLb5XhlO1238D9x7DZyuc4a79eycltai3SLnhDHhnZIe0GSwZSofW+bKAWgZyFi4AKM udWmD3/lfV1ROGi5ERHqYecw3dYlIgiVi+bHZTJ2VZ+jOvZgOzYmnkpNkucTslV9Nygrut vGyXZR58lww/H8nLrrtdw1X+YRiuybA= Received: by mail-yw1-f175.google.com with SMTP id 00721157ae682-565c7399afaso11712147b3.1 for ; Thu, 08 Jun 2023 18:26:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686273968; x=1688865968; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=PmbbTemShg7dGpCJymh8ioCm9PQPk/gLwHXF7mQ6jbw=; b=d+dKJf7MYgGkbogFV5iJ0in/u8QWUjYLBAFmQ7ygcUpoTtYR38tV8VjHHRx79+tqC1 xHAi8VEI8udjhn0i6NXjlbZGM0curW/G09YDGBDO7uNeymgEDUutuEhY67dcv9e1JC3c Kr/lYteRhi8EZZ4ON3qgr8IZYbs22sar5cHF6pJ+hkjtIcaYivcyvRaJDrPKAoQHoxL4 78cmtQoJihNLnrWGPJ0/wzQ+Ft29PFAH7mols+/0p1qq115e8foXt7EV0WK2TTvZzAlK WTFccNgXKTckPziGTMxd+vfG4KkV3xlgmbe2/Bx5rx+kPXmMSxor6N0wIGRCDIqsVdI4 znJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686273968; x=1688865968; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PmbbTemShg7dGpCJymh8ioCm9PQPk/gLwHXF7mQ6jbw=; b=FAadle8gnUUi9OoNjhZo/bq9KFIHYgBkODYGFzU+XI51yp5gkMcfwtgcFt3JkJ0pTD u/JIJe1yVW4RkdFwVYbfrI/0mzSlSvf5Y5nk9Xbnf/RTS7tynhkw4NvU5pxV4MQ9NMLc afHl/9BLT8FC/OnMwpLOGT7D7HWaLQGN95ZZAroJKP88dl1TWUEJUtovY3BANMmIS7P4 QmCRwYua25/zPT5MBB7p6BagxwhOKG751fKZz4iT6NkqpXq04Q4xDLcAHPYfLZJasBg7 +A+d5WnWLXnNL18bOhfWJViejgtQCriTXMRq1iHtAHVhMHRSF8Mx00WekULjsH6EdO7D qIMQ== X-Gm-Message-State: AC+VfDwymYsh1QGOsSMUKa2tppA1kHx/7ZSAayW4fj2PqykSd/O/odlj jMrPUDMvi6DcRKalUvyN2gEHSg== X-Google-Smtp-Source: ACHHUZ7XTCguPhOybQY+LH+ypWpoP0kYFfabibD6+mqWrxXo7bpsKAzEBYkvZOCdhGRiVh3fVLiw0Q== X-Received: by 2002:a81:5b89:0:b0:560:eadc:3bc9 with SMTP id p131-20020a815b89000000b00560eadc3bc9mr1402433ywb.7.1686273968353; Thu, 08 Jun 2023 18:26:08 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id p82-20020a815b55000000b00561e7639ee8sm296391ywb.57.2023.06.08.18.26.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:26:07 -0700 (PDT) Date: Thu, 8 Jun 2023 18:26:04 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 15/32] mm/userfaultfd: allow pte_offset_map_lock() to fail In-Reply-To: Message-ID: <50cf3930-1bfa-4de9-a079-3da47b7ce17b@google.com> References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 719A8C0020 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 3go8i8gx1ybp63pds9bbgibcrjm9hra5 X-HE-Tag: 1686273969-291014 X-HE-Meta: U2FsdGVkX1/bozcB7+7JZ2k/2+vmWq7K2k/El3mAQOUj2hDj1nTnoQMZ+lXRm+0GaCXq8OBtWnZZQpryLKk3QkKbmr++jrh1ZlZEDpBYPuW8slm1pmxadXeRz5HKs+7+9CDm99mPydOfCElSrxB2murjFxwYQ6kVWAM33VCmNBPbDT9jNaPZVxTepFCn5pPczX7b+MnJoU+dwk1BlmsDcAO10gFNFW7yZhTcK7Xmhh+JRYN2yotuhLApDnhoT2eRMjZiQkDfWIm5FrMPzTNTN2Z3b6jptywviAv2ABe9Xv0bC2/T05WbrCphwb32DWMHenBugBP3WJtCD9gd0ROgQpqpG2zhinMqiaVg4roqKwDDM7cdEcrjyrlQFGvIfPhK96bLCGMfJN5muAb9pY5sZbEZAW00/y+0cyNcGq2mz/+I2hkcCm+rM0gkSTz6PtknyIQbLywmqGElp1tsJsz/ySRKLV5cEl2vi8AGIq6yQw/zhHe6Cgoya0IimYInH814ffxmLTeAy9ilSiaVl+KFRinmImv32tCpQjqYxu9dLGhuoU6By/JFcegwwiM8/QvQo3UQ2EW3URA1Gpv4C2RGMwOli31gwD+1JGrQZ21uInbXQqgiq9ZioMmq5sHktPX955LX53tFfLyIAgBW5ZnKKl6d9ITjSOxd5KjbJ0Yxqw5ue76lc8up0VeC7XknCXamrTpVI6LTMEKgkn8pupScOLDSLlYbzTfpTXUQmaIn+mud3ub3fCzYgCOsjmd5ZsPu1b+izve+Re6zTNs3aq2TtkK0r+ITqKMpHetTajxPm/3en0z015b9ZffswQjb8cloXu/o7LGRK0sHQlwSUx9qm9y5gOiBN85ZGdsQIyhm8yO6dl393C2ZR94JF2qorQXbCZOv1NQ7XyGQ8XZd9NkQ+DFasaKOKDg1UYyim2rMXm2WsAqd5sSfokwIWwrTlfPnYiaQTPHbSyowyWfvisr eFws0JX8 QwggOy8PMF/jplN+s2LMoFozFuR/PocpGkf6XK2Dmsc/r2jkI07XwXTDx+FLl53lempC0IZaava3e4dnCLGlkU9D98NUCGkUO5k1/QWQoSBQ9eYvuJ8MeUQjUOpogBfFwHw9g1bKcjNXK1uqESZpeGWtx+/+uz6baWO/7xMo768khQeB+ZpV0mnbOvR0SJ+yHx6NcC8PnU/2KMz8WyllfKuwRB3GNe6UiPu9JgKW3bJlJ/CpWTgorT8LtOO13WLO8qkKcjpCBHMbVlmvi4NjaGDswtEChesjkfdyKJh4q92kj25PBxfqOmlz3a/2ByqC7XhN8NfiwLPKDDpoXgJWvtomJDyK6QrSGM5eFSlxwFJ5Sg+btmShyTqCl0aJF3rLSrBzyr8NMCyYU4tEkzAYVInibBSaNFSOGtUVXXnGpjfMvVgmgLNa6/+vCHg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: mfill_atomic_install_pte() and mfill_atomic_pte_zeropage() treat failed pte_offset_map_lock() as -EAGAIN, which mfill_atomic() already returns to user for a similar race. Signed-off-by: Hugh Dickins --- mm/userfaultfd.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index e97a0b4889fc..5fd787158c70 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -76,7 +76,10 @@ int mfill_atomic_install_pte(pmd_t *dst_pmd, if (flags & MFILL_ATOMIC_WP) _dst_pte = pte_mkuffd_wp(_dst_pte); + ret = -EAGAIN; dst_pte = pte_offset_map_lock(dst_mm, dst_pmd, dst_addr, &ptl); + if (!dst_pte) + goto out; if (vma_is_shmem(dst_vma)) { /* serialize against truncate with the page table lock */ @@ -121,6 +124,7 @@ int mfill_atomic_install_pte(pmd_t *dst_pmd, ret = 0; out_unlock: pte_unmap_unlock(dst_pte, ptl); +out: return ret; } @@ -212,7 +216,10 @@ static int mfill_atomic_pte_zeropage(pmd_t *dst_pmd, _dst_pte = pte_mkspecial(pfn_pte(my_zero_pfn(dst_addr), dst_vma->vm_page_prot)); + ret = -EAGAIN; dst_pte = pte_offset_map_lock(dst_vma->vm_mm, dst_pmd, dst_addr, &ptl); + if (!dst_pte) + goto out; if (dst_vma->vm_file) { /* the shmem MAP_PRIVATE case requires checking the i_size */ inode = dst_vma->vm_file->f_inode; @@ -231,6 +238,7 @@ static int mfill_atomic_pte_zeropage(pmd_t *dst_pmd, ret = 0; out_unlock: pte_unmap_unlock(dst_pte, ptl); +out: return ret; } From patchwork Fri Jun 9 01:27:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273080 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADE2CC7EE25 for ; Fri, 9 Jun 2023 01:28:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 461A08E0002; Thu, 8 Jun 2023 21:28:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3EB1D8E0001; Thu, 8 Jun 2023 21:28:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 263CB8E0002; Thu, 8 Jun 2023 21:28:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 146108E0001 for ; Thu, 8 Jun 2023 21:28:00 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id DF4CA1C7695 for ; Fri, 9 Jun 2023 01:27:59 +0000 (UTC) X-FDA: 80881473078.16.BB0171B Received: from mail-yw1-f173.google.com (mail-yw1-f173.google.com [209.85.128.173]) by imf18.hostedemail.com (Postfix) with ESMTP id 1779D1C000D for ; Fri, 9 Jun 2023 01:27:57 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=OyAJimk3; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf18.hostedemail.com: domain of hughd@google.com designates 209.85.128.173 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686274078; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MyPFy0gUSdpV3ibLKHHJg/J/UWoL9nTKOb5gbs61iKQ=; b=QiFUGkoICGkHCkwjR6US07s0ePIE17pxrN6DkILlzFRoNC9mM2zYPkxSnikyBrAbCO7t/L yNlp2UCm4y71sH4FbnH8y/l7zTkE2e5dPxGlrI0aYP304Tu1d+z40EvBTPhjUqOkgtgmMn hFixl3EShPjjLy8vebZ0vazzJAOvxNw= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=OyAJimk3; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf18.hostedemail.com: domain of hughd@google.com designates 209.85.128.173 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686274078; a=rsa-sha256; cv=none; b=m/8twCNaQhK7SQhWiRP2ucqvvVnRHiXNqA4ncKFSjCJ0Q6cIdFH/rqirFhfoCeHUz2aEIN M0R9kmos3VhrppNiw/Y6SUIrbRaoi7Oswsq+vNJtETV1ct1B73IC5Opl00vZEDz5Ys5uRu 2+KbrQzN5IhxSxnpn2dQr2WiQzvqnU0= Received: by mail-yw1-f173.google.com with SMTP id 00721157ae682-565c3aa9e82so12131027b3.2 for ; Thu, 08 Jun 2023 18:27:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686274077; x=1688866077; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=MyPFy0gUSdpV3ibLKHHJg/J/UWoL9nTKOb5gbs61iKQ=; b=OyAJimk3fc+/Li5me7PIcu882/+lDPwqec67fOflTDld8VWvm0ZySNcdoJgLTrD9rx WoOLU/XWyP4MN+Eoi7KJZTv1R6ExtbK2SMofkB2PIt1uKNhRHeS+xx4hN9Wks9SB/Hkk 4gzmFYarU9qov9Uzj7o9h1zLdkAzYRq/b/kKfVItrvxfhKgU8iMFFvE5pQZ3kgV2up+L hNFZoE2nbmMWdsnJ1XbGUBg6gK5gwmUXxa01pub4RN5ECsXTRQyDMvk6goIXaQohEPKU wCMnLqlIXUxUsnIlKrzfLeDs5rhQwhiPDht5XmK01inAm6KH0sUkni/+00j7bluhyFK7 RwQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686274077; x=1688866077; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MyPFy0gUSdpV3ibLKHHJg/J/UWoL9nTKOb5gbs61iKQ=; b=QdIXhHuCE9+na7R0j2E2DyVsgSk2udFW4q5LPlRUcaf+9hoOEm6hQs/RgWWrnV4FMI CpzdywX/ilXs6vB+E4V0P3Eotk3yVzoz5/i0Vg2gTMBmM7Kttcrf/AAUWvxm7BkSOxxU A8HAGhoBgp3LwlWhRB0TGMTwG9uHHrdgE7+5axQwdyiE7t3LVinrNehnv4oZdl/qsqIu tgJAIRHNp8uVg1flr/mWC68c9yM6AR0rpxFV3oJ2lIH5HOEHXZXIQi5z9onvM2QCQvgG 2YStXfrWQZ2IvzqLTjKTLN902F0HYQ6AyRp/YCgYusHfveTHkvtJrWqrZ0Dun47pyQCV AyDQ== X-Gm-Message-State: AC+VfDyg9a0jr9+DO8EnJypiQx14GhFPpAKa/oviSi5t2ORIdpDofdtS A0JEdhHIfzQ6a9HV/z59MLHKAQ== X-Google-Smtp-Source: ACHHUZ5jGBBEQ5gswQSXjac3sdwb/7+VqiRoKupCeGhVPyqG0jNqvRoD86vqt+pEd3PQOO1HZrsBzg== X-Received: by 2002:a81:8005:0:b0:569:74f3:ed2e with SMTP id q5-20020a818005000000b0056974f3ed2emr1337210ywf.22.1686274077005; Thu, 08 Jun 2023 18:27:57 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id c205-20020a814ed6000000b005688ca40c4bsm291663ywb.61.2023.06.08.18.27.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:27:56 -0700 (PDT) Date: Thu, 8 Jun 2023 18:27:52 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 16/32] mm/debug_vm_pgtable,page_table_check: warn pte map fails In-Reply-To: Message-ID: <3ea9e4f-e5cf-d7d9-4c2-291b3c5a3636@google.com> References: MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: kxqdhchtwmzpmqeeyibdbf8qm3tckzfy X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 1779D1C000D X-HE-Tag: 1686274077-362586 X-HE-Meta: U2FsdGVkX1+zPKzQgDwAdWRh5cyHN5Vp3QyNbSx7pHm6nA7aOsYh2NqmLGnXdJOzrklY5/EfMC0XQ/VyOS49Blvj0kRo+k/IvJcYOLuA7tluZhrYCnjRTfGtPFLG/3wSGYSWtx4HR+C9j+lbef3XfbIGUzpXt4iosMmdTJlG/l1eBlyW1VfZk09DEZF3tnWokDsxfzvEEcHchZN5kHhN7Fmf/FsyFHtqFT55S/LOSGe1MhToG8tnRTMf4AK0rDOZ0l8B9jjn9F5S6iSEqq+PMe9jyXV46O8IQeL08jw+/YoLOU5ZEKRKCEXdXpXpYUWeCEQqtLYjqcSy8Xxf0+ftElguQVytXn8C0eCyDYhUloz56udUu9cHfHTTGxgLa6KZeJVsvd+RGjhPMJNKdHKZ/DopfaDgoOB9Sl5FcHi0HyCA/fjetv3t0m1fd8EW62nzWAGxZfjgolpYnJZQy+CWYlYtvTgbQH+9cqhG3IJIDhcpEvpPu+IvcMypv1/0N8Cg47TnrD41e9qNEnMGM572ZFXA1De+EmI3RwIEQn5sLbPzhDdvx/HExmYKxF41/bIwQPHTdO3Ymv6lbn91ELrtuZzzTNoLSRVMb1QkZM5YyCWRuyIv2QVheG/vsKrCm/E/Yvy94Km32oZvkkYMAIQrX+pVbyE4hR3XfEED1xrgzdFbE4B1lnEgi2R0BM/tWsalF0j+6Jovf3fI3e4ESmPYcqflt3S35fqb0OeL625Tsd9yA3ii4TRACjmk+rFMMisB9oADqjnljGIrN4G5+k5QppmmHFFLv56Qo7umrXVy4bDJjzFbQOhH10qhIwSKgFRF6Sun1FbzY13Krpce34jEayobNyofBehEr+veFr0bUmFYCudMKvIcNrFUxzvrAoZlZPJPyVQhVMj6Ykzp0lTOOHZbk/I0Z1DVpGYxPOwBfhsumpllrKlSaG5OvjM9TrjEr8R5j6eWZUm4hRo2w1N j3uPLugk sZvdhjgaZVoBbfSNXnRSN713pRGlkerRSGYEM95Trq+JcMtQNAyO4QcTYv/vGei7h2qE0raX/WmBPppmqHd1gBpiXzTTID9jTOVqR4CmEU04rHT7AzjGKO4ZLO3L/oNd7AZuNQHILIkdF2QvyUPBSBlOtuikKueoGrxWyskagUi9LjdsFnmpsYJ0DHTSdZpYXRrVHYFEhV7/EcpdwhczoY53sp/sYmkP94R5rSw6WO6gQjjSwlmPyr3EPJ4c27XA8Flse/2rFYmiWeHRzm9Bb2nXKQRgsYp+BUSN5WOXyZO7e5PzYnypgNuYXDms7AJ2oB7M2xbVtbKTi450QQGetXYGXTzanCc5iShutidu5U34qe6WMtflac2X40Uu820qjIb9/WCWHZsJeodKSawUBqAVlFY7ufwsp1Ffj9UYf9jtAbboI3rO2ihyQLg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Failures here would be surprising: pte_advanced_tests() and pte_clear_tests() and __page_table_check_pte_clear_range() each issue a warning if pte_offset_map() or pte_offset_map_lock() fails. Signed-off-by: Hugh Dickins --- mm/debug_vm_pgtable.c | 9 ++++++++- mm/page_table_check.c | 2 ++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/mm/debug_vm_pgtable.c b/mm/debug_vm_pgtable.c index c54177aabebd..ee119e33fef1 100644 --- a/mm/debug_vm_pgtable.c +++ b/mm/debug_vm_pgtable.c @@ -138,6 +138,9 @@ static void __init pte_advanced_tests(struct pgtable_debug_args *args) return; pr_debug("Validating PTE advanced\n"); + if (WARN_ON(!args->ptep)) + return; + pte = pfn_pte(args->pte_pfn, args->page_prot); set_pte_at(args->mm, args->vaddr, args->ptep, pte); flush_dcache_page(page); @@ -619,6 +622,9 @@ static void __init pte_clear_tests(struct pgtable_debug_args *args) * the unexpected overhead of cache flushing is acceptable. */ pr_debug("Validating PTE clear\n"); + if (WARN_ON(!args->ptep)) + return; + #ifndef CONFIG_RISCV pte = __pte(pte_val(pte) | RANDOM_ORVALUE); #endif @@ -1377,7 +1383,8 @@ static int __init debug_vm_pgtable(void) args.ptep = pte_offset_map_lock(args.mm, args.pmdp, args.vaddr, &ptl); pte_clear_tests(&args); pte_advanced_tests(&args); - pte_unmap_unlock(args.ptep, ptl); + if (args.ptep) + pte_unmap_unlock(args.ptep, ptl); ptl = pmd_lock(args.mm, args.pmdp); pmd_clear_tests(&args); diff --git a/mm/page_table_check.c b/mm/page_table_check.c index f2baf97d5f38..b743a2f6bce0 100644 --- a/mm/page_table_check.c +++ b/mm/page_table_check.c @@ -246,6 +246,8 @@ void __page_table_check_pte_clear_range(struct mm_struct *mm, pte_t *ptep = pte_offset_map(&pmd, addr); unsigned long i; + if (WARN_ON(!ptep)) + return; for (i = 0; i < PTRS_PER_PTE; i++) { __page_table_check_pte_clear(mm, addr, *ptep); addr += PAGE_SIZE; From patchwork Fri Jun 9 01:29:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273081 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2102C7EE25 for ; Fri, 9 Jun 2023 01:29:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 46DB68E0002; Thu, 8 Jun 2023 21:29:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3E9ED8E0001; Thu, 8 Jun 2023 21:29:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 264678E0002; Thu, 8 Jun 2023 21:29:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 17EBA8E0001 for ; Thu, 8 Jun 2023 21:29:30 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id EB0811C78FC for ; Fri, 9 Jun 2023 01:29:29 +0000 (UTC) X-FDA: 80881476858.03.8444AC6 Received: from mail-yw1-f171.google.com (mail-yw1-f171.google.com [209.85.128.171]) by imf23.hostedemail.com (Postfix) with ESMTP id 1E42F140017 for ; Fri, 9 Jun 2023 01:29:27 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=1iKYEdXQ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of hughd@google.com designates 209.85.128.171 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686274168; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wEXGzvow+PeeAaek8TvLjmsJIBTHvkDkqgMD7Fw0vH4=; b=T6RT8DiMQRqJ3933J7oOp89MOEwA20VuGeVAOyR+6PfJEVEcMNIus0z5Hq7OIWgmW2p8T0 rUXDoN41bkNwGtW+IYmkC5ehRPfB55sRrG/cvLW4oA63q5jFzpoT/w8KewKIZg7VWXg/cl e/TaGWFX/FL/uJXwOglcPmUaTyoaF1Q= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=1iKYEdXQ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of hughd@google.com designates 209.85.128.171 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686274168; a=rsa-sha256; cv=none; b=40sKwftqpzBro1Na1Ff+qEwZ5/7DaXnW2j3nCGfyIYaw6Wm/c+r0huf0FG8ZJxNfm66owD W6/Ts4hYQMxzCr47z2MoUON7YJC5fxIHk+F2BHDbn/XNilUvX3TqLJZm/QunIZv0jkfGQM tV6ZsuH7z5bapZRriXZp5GkEwJOHtM4= Received: by mail-yw1-f171.google.com with SMTP id 00721157ae682-56896c77434so11393267b3.0 for ; Thu, 08 Jun 2023 18:29:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686274167; x=1688866167; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=wEXGzvow+PeeAaek8TvLjmsJIBTHvkDkqgMD7Fw0vH4=; b=1iKYEdXQU7GyBWXXdTJ/eTNQaKjM8A5uGEtCPlSPe50PEEnGNTaKjIFQd2nLJWOfvh tPsKmHoib44kj+JEUNUeUUirOK6gcFjvoUWHBf5F5sWsiABdMBGtAthxe2WU9qh7noM4 Zh/iyBIZbzHd4k5pq9B8glalrk6L5lYQzkgizx3QlltAhP4mUsNeTxTjLywFtxVFGNvZ jIQGXg/oBKKP81SqRx4Caj2rWxiav0FLHH/NmIS/432ybPMUtKQi+Yq1y0GkDZpiwhGe kNBiMEtti/dkcWXs/faQIjCPcs483ukabcMmUPTpwrdxz/Ln2gtVt+7567CoyLJvwmZM 3ejA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686274167; x=1688866167; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wEXGzvow+PeeAaek8TvLjmsJIBTHvkDkqgMD7Fw0vH4=; b=fILGNgNX65I0qsbqTuRY1f8UmfFrUdWtwjvO/QD6cDuuMbbmq4THmsrUIq1qSaOfCf N4ZTao2kYRuq02taYwcZldKCRJHcjdblTv4gD2r2GIGPQTBpXfA1TpmwB5OIPLXI1ifL jY7C3WYwWjVQ0wH1sM8Ea4pUeAYtvHEYv28LuGKw47Dk68jWI7PRCjQ24c9vVwhyNBf3 eO75wOOgMCZgamxFhpXrZxH9Gii9SvmuV4zXgYi0rndL4tv39yQqLTAUb7MoC8cPsk7E 8vQzaQZjvpTE8cutAoDwdJ1I35oACvUhvUtjdasw1JvLJRo1+Ov7nNT0Oy7pzfQyR0fl F79w== X-Gm-Message-State: AC+VfDxCPz00kAhvLSQkQtZq2jPq6lj6C/DeJT23+SQ6h578mMrDFB3O joMOI4G9MqE49L3Mi3aGQt+qog== X-Google-Smtp-Source: ACHHUZ7s1itVLKfUVkBt3mJsfUoKC8ShOR4DXlaWFBNzOobINfIUYkdb8mmHGFUq5qbj4FXPU58kgA== X-Received: by 2002:a0d:e8ce:0:b0:561:be2a:43f9 with SMTP id r197-20020a0de8ce000000b00561be2a43f9mr1268402ywe.41.1686274167029; Thu, 08 Jun 2023 18:29:27 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id i130-20020a815488000000b00561e2cb2d3bsm300434ywb.23.2023.06.08.18.29.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:29:26 -0700 (PDT) Date: Thu, 8 Jun 2023 18:29:22 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 17/32] mm/various: give up if pte_offset_map[_lock]() fails In-Reply-To: Message-ID: <7b9bd85d-1652-cbf2-159d-f503b45e5b@google.com> References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 1E42F140017 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: fsy9gd5hi8zih5jhhbo34qnsqs76ahnh X-HE-Tag: 1686274167-710937 X-HE-Meta: U2FsdGVkX1/4ECAba2kXje160VQ4VQJcusdinGDWNp0gqfmcRJ73v2fdMqrzPLqI9tlLUS3NIMgQGeZQzTfOnrmCny95DF/TTyAckuxWeN2/O0A4g/xfZunnbCHuJELRbOoZ1hsvGiEquDOjjHD1DRSS2cWXmJsc4yAC2C1GTO1YaL5V9VUEz8a5VhQ+YAYME+1cJxzzb5QTdUbLz/N9mFXnZLkwKkKsGHbrfiRKmfaovithPk7n06g90mlv6kKXLkbEaktMwbFgJET6kAWpxBLS9UZ9D//qwVRks5SfLp3hlrCLiu9k1geXQREWvgc75SQgvH4N5Tnh6ZYNl4ObYpU3utPmHgYox5W/NmNqxVjpGrbJ5EXXs1ApA+GjjC2oCpsVv5pzmoEyNc/L3a4M4jMF7rnamoZRAJrS5M67dMmxS9/bl8dk8UtW2Mbqg6TIKojxq4Mr0ywEUsn4PjrgrDVzpmRFKU2c5eJpXHJd0sbOcAlsxsy8qQiyjuIM4ovbwCe09Trg5e7MICAd6J+SbcmiDz6qDm9p3FFsviQCESRdQEV63k4C5nCPyotrE+Q7Nx6J/mNneZ6L+lQTcFGo0H2QbbrTNV3jzoNuPa3EkKgrGPizPuJNXb/LwGFEB2F+hvw3qHC2kgZCUzfUqKc+H5kdDcTbJui4qrZi3V48jUqoI+JQW4aFVoZxrYP6OJoFUO/v/6aE7OwyuriMOdKCQE+T4q7TCD0IWYYAu1Tg78opv9PQFVIDwFN1CHL3/xVayJnc/3KkW35ghPua2WNExsbi40WLb7E6zxwfnGyoFIRND5fZV9oc6GR70F/1FoRDVfX9pFj9gXD7xOJBvbVun1NXOjTuIZ3ErL8Ktz5AcZ05qiMECWitO9yVJD5HgLjQxHFnD0P9vlu+VE8VtYpZvwi/B7u/4wNlTmhs4eS/4XskBkqYHW1AxoBhSWV5QH+XrvTvUWYykVMP1BksxBS lt+WDvDU Ldkp4LAfb+mUGhW6cGpEdVclKYHmalp65xithzWxJ6ReKsbw8BLvP6If4QmfNWe6MZevFredLdl6nIj16J1orodABt7/1fIvooK0crqKiL9z476b7jX6iuwwaQPGUDEMVlNLq4v3j+WVZClt948uHbfI40XdGqPhNGWsQ61DXfxukrgOXa4V2GWn5PTpF5TBCVNAGIpeGqKS5S/Nt+xGQUz6jxXn+5CV4e0Ec8g/CiDnx9Zr3MoCu0jLaoK1btYimWZwE/7GwoZOuUX288XXZA7GaUAQkEil0QSqUDr0nd8KgU3slZelE2U5j27FTpNaEkvHPbgY/Bg9+089o2geu7p8HK6jByEwyky9FgYqL+xOpFLr1CuovPP/+w10afliKLO/d3gI4i7InvsdrQ20015ovvmkbDsuBS5rzr+332fgMxxs37zWY7tn0QA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Following the examples of nearby code, various functions can just give up if pte_offset_map() or pte_offset_map_lock() fails. And there's no need for a preliminary pmd_trans_unstable() or other such check, since such cases are now safely handled inside. Signed-off-by: Hugh Dickins --- mm/gup.c | 9 ++++++--- mm/ksm.c | 7 ++++--- mm/memcontrol.c | 8 ++++---- mm/memory-failure.c | 8 +++++--- mm/migrate.c | 3 +++ 5 files changed, 22 insertions(+), 13 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 3bd5d3854c51..bb67193c5460 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -544,10 +544,10 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, if (WARN_ON_ONCE((flags & (FOLL_PIN | FOLL_GET)) == (FOLL_PIN | FOLL_GET))) return ERR_PTR(-EINVAL); - if (unlikely(pmd_bad(*pmd))) - return no_page_table(vma, flags); ptep = pte_offset_map_lock(mm, pmd, address, &ptl); + if (!ptep) + return no_page_table(vma, flags); pte = *ptep; if (!pte_present(pte)) goto no_page; @@ -851,8 +851,9 @@ static int get_gate_page(struct mm_struct *mm, unsigned long address, pmd = pmd_offset(pud, address); if (!pmd_present(*pmd)) return -EFAULT; - VM_BUG_ON(pmd_trans_huge(*pmd)); pte = pte_offset_map(pmd, address); + if (!pte) + return -EFAULT; if (pte_none(*pte)) goto unmap; *vma = get_gate_vma(mm); @@ -2377,6 +2378,8 @@ static int gup_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr, pte_t *ptep, *ptem; ptem = ptep = pte_offset_map(&pmd, addr); + if (!ptep) + return 0; do { pte_t pte = ptep_get_lockless(ptep); struct page *page; diff --git a/mm/ksm.c b/mm/ksm.c index df2aa281d49d..3dc15459dd20 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -431,10 +431,9 @@ static int break_ksm_pmd_entry(pmd_t *pmd, unsigned long addr, unsigned long nex pte_t *pte; int ret; - if (pmd_leaf(*pmd) || !pmd_present(*pmd)) - return 0; - pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl); + if (!pte) + return 0; if (pte_present(*pte)) { page = vm_normal_page(walk->vma, addr, *pte); } else if (!pte_none(*pte)) { @@ -1203,6 +1202,8 @@ static int replace_page(struct vm_area_struct *vma, struct page *page, mmu_notifier_invalidate_range_start(&range); ptep = pte_offset_map_lock(mm, pmd, addr, &ptl); + if (!ptep) + goto out_mn; if (!pte_same(*ptep, orig_pte)) { pte_unmap_unlock(ptep, ptl); goto out_mn; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 4b27e245a055..fdd953655fe1 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -6057,9 +6057,9 @@ static int mem_cgroup_count_precharge_pte_range(pmd_t *pmd, return 0; } - if (pmd_trans_unstable(pmd)) - return 0; pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + if (!pte) + return 0; for (; addr != end; pte++, addr += PAGE_SIZE) if (get_mctgt_type(vma, addr, *pte, NULL)) mc.precharge++; /* increment precharge temporarily */ @@ -6277,10 +6277,10 @@ static int mem_cgroup_move_charge_pte_range(pmd_t *pmd, return 0; } - if (pmd_trans_unstable(pmd)) - return 0; retry: pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + if (!pte) + return 0; for (; addr != end; addr += PAGE_SIZE) { pte_t ptent = *(pte++); bool device = false; diff --git a/mm/memory-failure.c b/mm/memory-failure.c index 5b663eca1f29..b3cc8f213fe3 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -414,6 +414,8 @@ static unsigned long dev_pagemap_mapping_shift(struct vm_area_struct *vma, if (pmd_devmap(*pmd)) return PMD_SHIFT; pte = pte_offset_map(pmd, address); + if (!pte) + return 0; if (pte_present(*pte) && pte_devmap(*pte)) ret = PAGE_SHIFT; pte_unmap(pte); @@ -800,11 +802,11 @@ static int hwpoison_pte_range(pmd_t *pmdp, unsigned long addr, goto out; } - if (pmd_trans_unstable(pmdp)) - goto out; - mapped_pte = ptep = pte_offset_map_lock(walk->vma->vm_mm, pmdp, addr, &ptl); + if (!ptep) + goto out; + for (; addr != end; ptep++, addr += PAGE_SIZE) { ret = check_hwpoisoned_entry(*ptep, addr, PAGE_SHIFT, hwp->pfn, &hwp->tk); diff --git a/mm/migrate.c b/mm/migrate.c index 3ecb7a40075f..308a56f0b156 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -305,6 +305,9 @@ void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, swp_entry_t entry; ptep = pte_offset_map_lock(mm, pmd, address, &ptl); + if (!ptep) + return; + pte = *ptep; pte_unmap(ptep); From patchwork Fri Jun 9 01:30:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273082 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9BA56C7EE29 for ; Fri, 9 Jun 2023 01:30:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3288E8E0002; Thu, 8 Jun 2023 21:30:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2D87A8E0001; Thu, 8 Jun 2023 21:30:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1790A8E0002; Thu, 8 Jun 2023 21:30:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 092EF8E0001 for ; Thu, 8 Jun 2023 21:30:56 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id C6FAFAF074 for ; Fri, 9 Jun 2023 01:30:55 +0000 (UTC) X-FDA: 80881480470.16.545F584 Received: from mail-yw1-f177.google.com (mail-yw1-f177.google.com [209.85.128.177]) by imf07.hostedemail.com (Postfix) with ESMTP id EBB1F40019 for ; Fri, 9 Jun 2023 01:30:53 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=gJAMMyka; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of hughd@google.com designates 209.85.128.177 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686274254; a=rsa-sha256; cv=none; b=RI361BZo1xgAVsw4WC+K1N1YvZ1dCPKxiybYzQ/v+Iq0F5kvIUdIEQFV39Tzq7lJZtWVhC 0+nI+ugBS0lyrgEF25b3Q8asKIOglgPzjlrD7RrXVgURC8Ck9O4hTTyMb9b7T8WSrSgAiL xOu8gg6+AQnyO66Vfiwpet03ivlcelk= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=gJAMMyka; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of hughd@google.com designates 209.85.128.177 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686274254; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7GAfzbPNj/2rvKRevzngE24N05kydgFSIoMtFyLCrv4=; b=y4nzi4lm4ZjG/IoRcRh8rHNJjKkxmS5n3f3oXQF+eThhq0rT4nJBhOvRM7EA2BA8VpiX5W pGojoeaRAxI/jw3AqpeoniOf9QNzNnZVDIv2i3KEbHaHt6PUB0sq1mjbVHm6go7oxMaHyH ngJRTQL0foDIQC3FW8MPUQ+igGbJ/nI= Received: by mail-yw1-f177.google.com with SMTP id 00721157ae682-565bd368e19so10851897b3.1 for ; Thu, 08 Jun 2023 18:30:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686274253; x=1688866253; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=7GAfzbPNj/2rvKRevzngE24N05kydgFSIoMtFyLCrv4=; b=gJAMMykaIEU9PkjyCIdSz3ODq0ftmm75epqTVf1DDwNHXYM11vOXndav/A+j1kRFEc 9zEcP31A1kA13aWMOjns1R75xfAWzNDi1gHKk68ewQgq0whuvZDJ2IOLlB+uSUVd5Hp8 svpukbwsNHcvaTpeSeeB57vhMGo3aoLUGSbQC4vDY82UJZYh6UFZthAmKEZpoO6CrQsj zIurR6IMYJtElj3ek8vexr06I8pOIvm9uDnN6jHks/XrVIB5jV9tJeQ4OkFBFlzOLvfn TFupD/LP2R58aC1XHY0D8ruvxKPXB9C8uw3aB2kdOGhPoyncXso5dO85F2PIfWP2s5CT /ipQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686274253; x=1688866253; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7GAfzbPNj/2rvKRevzngE24N05kydgFSIoMtFyLCrv4=; b=AWclyB37ba4uUf18vfF2DNc7hIvWL4aPPnX5F/Qdq4D2cw4ipLic/hzSMtsVoLdIM6 IqHiyW1+eepcjZB4kAfLfraAHpP6+9YaH8DzLvRdUjIPQzRDF/3AR4qlkrvcrqQWYRta yyOg5SkXWdFTNNU6BwbenZwEGSd5ffrmcJeBur+J7gyCyuovnQmHdUKUGan8e2ifXJkq JeEgGfIvpTZgmNzhWOt8Gg+YZ9ybhy7bablBpuY+oJqfDQ5A4sK8C4HQp9iVpunxKEVu DN77af2URMuhHcTsPZTwPDzmDNQ76nulRyW541NIi3Ty6svR3f1FfMc9m119Vb5ziP8u UuyA== X-Gm-Message-State: AC+VfDyrWGNz9wPMq3tLYnW2zJxFX+jUyhUiYLfugDWeOIPA2ObLRg0V hhXuihlFTXR8g3j++6scdx1I2Q== X-Google-Smtp-Source: ACHHUZ4zYAZPyewfaoixPaut1Ay+u+IP9P5gVJt0aeUL3XpkhLRU1BODxkde89jUMQkygUDst/++Nw== X-Received: by 2002:a0d:ea8b:0:b0:559:d3a0:4270 with SMTP id t133-20020a0dea8b000000b00559d3a04270mr1322987ywe.34.1686274252865; Thu, 08 Jun 2023 18:30:52 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id y206-20020a817dd7000000b0055a881abfc3sm277355ywc.135.2023.06.08.18.30.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:30:52 -0700 (PDT) Date: Thu, 8 Jun 2023 18:30:48 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 18/32] mm/mprotect: delete pmd_none_or_clear_bad_unless_trans_huge() In-Reply-To: Message-ID: <725a42a9-91e9-c868-925-e3a5fd40bb4f@google.com> References: MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: EBB1F40019 X-Stat-Signature: 5g6ckapysswx11zzyewh74z9pa3qs4pc X-HE-Tag: 1686274253-333391 X-HE-Meta: U2FsdGVkX1+DfPJJrM590xCiJav1fQfODU1Yh6SBuWHQB+tCnR6ESd2MnU8a2ngj4s3SIAk8NM3PmLS54bFRJspJ+2svfGuW4Pi21txVkYn/t2U44A3w/0uH+QCPLO7x1LlFk3u/Cc4okeirGTEFs9dfyv+eTdXMJNk+czGeY2U9NgjMBebUV5Knuruhb2y1U/xFMVRpoH+YLsNhG6LHxRZqST99vv3PfikZ4xpmdGdT7yFHdKlXUQ0W6O3wvme86dmxfoUz8sX+qLMkpTFNVoyyHd7N3iIqLXlzvkCbj660iqawmgIdynZI1viqpeScLaMj117+wxHDLeXoER4imFSVOG6862cpE3f+oEHiXgkxbMyuQXhxxozkoYF1w9Eqw7Bw7e3ilsXhZqErsHeXPZ1RHBhwgyEbV9bj2C0GWjwS+bJ/DqLZxjSiK+V043enkjf86ZJx6x476/sL+YQNHgr9yaWa1PHLafX5XeyjFkJyRbPZqtpO9RtRk2Dq4J5rdeUkS1bw7BdbbJaqb9olSKyDMjZbVvyc9kl0Nso8KCwC/n4RWEk/e7euGC3K9juY9K36BmSCko7UxplxCSLkuAgAInjOGl5HfW8a07HQwUY5nqysXXPY8lS/pnlbja5ZZo20LPHIivMyvYYT6gqYdNWtv9ex/7dAwjS0HFjf22JEnXOgq0pKbg21vJeDiDZSWI7xUz7YkodZgUSR/lTH2Q9Ad3rK+swvBxb9JTjJ3HYesiMAY+RMY/Y67LPxZtxHVD6pH4yXUDnzB4aXj7yI0TqvIA+kOJpILgLCId1un+pkjWPnGL7ZcBBmsPFfHWPhgF1c1Pq9L0vEg/fH5h8b/Xp04/XguBqwCOvKJzavHsu1O4XP4h/uFKZo2UgmMYQhP5vquYcFNlmRO++xndqnpgTQ8ZNl6qgXm1lfgL6r7ARMu2JD6oFYqE6l8N147Vr4+VJMpTX2r/z6qobqCx4 hURLkmxD QSlFkfMvJCcEtIiLLUAA9PCgLCG/GzO9qPQr+JH4yOC3t8voKd8Sxr0ePkDZ+vapcpKWwfkflXh8uouQM7amlSnW7g2xfPyPX7zotXpSoxjJhvvt8oxTPgbopN3ae6I9I4Tfnk2zvxE5GwfjDEt2Q/7rXNjKBz4bOzq7qxvmGuB8HHDSOmObPT2D8Lj8Iv2Ol4cRE6MrOvgPRkUEctE3LwXG9qAaQAge1mHRC/FNN4uImQMxI5El4iXfKrDQ9pklxST4xxSZsO1pkFGpK15fipc1JppBHlrCnEI0xdS5cMEBJr3je9oyDM4ftqOKEembPDAxuBO71RDEZNLQYQBZeMDJt7tX01s7AUwEQ2bHLO+WqafIm3oFMxnQKzRVfhuqNi/aF4QQ/JfwwlQRTxwDI29OrAVJg87b90aGwAmNdcjdPCHOFYrWE7S3GGQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: change_pmd_range() had special pmd_none_or_clear_bad_unless_trans_huge(), required to avoid "bad" choices when setting automatic NUMA hinting under mmap_read_lock(); but most of that is already covered in pte_offset_map() now. change_pmd_range() just wants a pmd_none() check before wasting time on MMU notifiers, then checks on the read-once _pmd value to work out what's needed for huge cases. If change_pte_range() returns -EAGAIN to retry if pte_offset_map_lock() fails, nothing more special is needed. Signed-off-by: Hugh Dickins --- mm/mprotect.c | 74 ++++++++++++--------------------------------------- 1 file changed, 17 insertions(+), 57 deletions(-) diff --git a/mm/mprotect.c b/mm/mprotect.c index c5a13c0f1017..64e1df0af514 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -93,22 +93,9 @@ static long change_pte_range(struct mmu_gather *tlb, bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE; tlb_change_page_size(tlb, PAGE_SIZE); - - /* - * Can be called with only the mmap_lock for reading by - * prot_numa so we must check the pmd isn't constantly - * changing from under us from pmd_none to pmd_trans_huge - * and/or the other way around. - */ - if (pmd_trans_unstable(pmd)) - return 0; - - /* - * The pmd points to a regular pte so the pmd can't change - * from under us even if the mmap_lock is only hold for - * reading. - */ pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + if (!pte) + return -EAGAIN; /* Get target node for single threaded private VMAs */ if (prot_numa && !(vma->vm_flags & VM_SHARED) && @@ -301,26 +288,6 @@ static long change_pte_range(struct mmu_gather *tlb, return pages; } -/* - * Used when setting automatic NUMA hinting protection where it is - * critical that a numa hinting PMD is not confused with a bad PMD. - */ -static inline int pmd_none_or_clear_bad_unless_trans_huge(pmd_t *pmd) -{ - pmd_t pmdval = pmdp_get_lockless(pmd); - - if (pmd_none(pmdval)) - return 1; - if (pmd_trans_huge(pmdval)) - return 0; - if (unlikely(pmd_bad(pmdval))) { - pmd_clear_bad(pmd); - return 1; - } - - return 0; -} - /* * Return true if we want to split THPs into PTE mappings in change * protection procedure, false otherwise. @@ -398,7 +365,8 @@ static inline long change_pmd_range(struct mmu_gather *tlb, pmd = pmd_offset(pud, addr); do { long ret; - + pmd_t _pmd; +again: next = pmd_addr_end(addr, end); ret = change_pmd_prepare(vma, pmd, cp_flags); @@ -406,16 +374,8 @@ static inline long change_pmd_range(struct mmu_gather *tlb, pages = ret; break; } - /* - * Automatic NUMA balancing walks the tables with mmap_lock - * held for read. It's possible a parallel update to occur - * between pmd_trans_huge() and a pmd_none_or_clear_bad() - * check leading to a false positive and clearing. - * Hence, it's necessary to atomically read the PMD value - * for all the checks. - */ - if (!is_swap_pmd(*pmd) && !pmd_devmap(*pmd) && - pmd_none_or_clear_bad_unless_trans_huge(pmd)) + + if (pmd_none(*pmd)) goto next; /* invoke the mmu notifier if the pmd is populated */ @@ -426,7 +386,8 @@ static inline long change_pmd_range(struct mmu_gather *tlb, mmu_notifier_invalidate_range_start(&range); } - if (is_swap_pmd(*pmd) || pmd_trans_huge(*pmd) || pmd_devmap(*pmd)) { + _pmd = pmdp_get_lockless(pmd); + if (is_swap_pmd(_pmd) || pmd_trans_huge(_pmd) || pmd_devmap(_pmd)) { if ((next - addr != HPAGE_PMD_SIZE) || pgtable_split_needed(vma, cp_flags)) { __split_huge_pmd(vma, pmd, addr, false, NULL); @@ -441,15 +402,10 @@ static inline long change_pmd_range(struct mmu_gather *tlb, break; } } else { - /* - * change_huge_pmd() does not defer TLB flushes, - * so no need to propagate the tlb argument. - */ - int nr_ptes = change_huge_pmd(tlb, vma, pmd, + ret = change_huge_pmd(tlb, vma, pmd, addr, newprot, cp_flags); - - if (nr_ptes) { - if (nr_ptes == HPAGE_PMD_NR) { + if (ret) { + if (ret == HPAGE_PMD_NR) { pages += HPAGE_PMD_NR; nr_huge_updates++; } @@ -460,8 +416,12 @@ static inline long change_pmd_range(struct mmu_gather *tlb, } /* fall through, the trans huge pmd just split */ } - pages += change_pte_range(tlb, vma, pmd, addr, next, - newprot, cp_flags); + + ret = change_pte_range(tlb, vma, pmd, addr, next, newprot, + cp_flags); + if (ret < 0) + goto again; + pages += ret; next: cond_resched(); } while (pmd++, addr = next, addr != end); From patchwork Fri Jun 9 01:32:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273083 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A038EC7EE25 for ; Fri, 9 Jun 2023 01:32:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3C3B68E0003; Thu, 8 Jun 2023 21:32:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 34C4E8E0001; Thu, 8 Jun 2023 21:32:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 214C58E0003; Thu, 8 Jun 2023 21:32:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 150BA8E0001 for ; Thu, 8 Jun 2023 21:32:55 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id E1FB31602C7 for ; Fri, 9 Jun 2023 01:32:54 +0000 (UTC) X-FDA: 80881485468.16.D660C44 Received: from mail-yw1-f176.google.com (mail-yw1-f176.google.com [209.85.128.176]) by imf03.hostedemail.com (Postfix) with ESMTP id 147F32001A for ; Fri, 9 Jun 2023 01:32:52 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=E4DRQK86; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of hughd@google.com designates 209.85.128.176 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686274373; a=rsa-sha256; cv=none; b=dWSxCFkmMQsEu9Fe9cdSjSLDGGadTVsk78NMmqR2ZlEEuvwsusXhmdZH/sNm2RUkXTfDSm t413uFphMv1U2EXG3RBjrOhNWL9osoTJeU9ahQ2/TrxivAZWq9AUkjb3TtgF4xw8h9Udbh iPcBaOVS4IhJtSmXs+FtJkRbHVnqBn4= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=E4DRQK86; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of hughd@google.com designates 209.85.128.176 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686274373; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dPoyGlkGpegFUTFofVDIHWRBddk7VxC7U3Cl7g+dttQ=; b=CHfO5MzYCokAKGRrmeNmThEbhDGCOlOJHpnyvMgKu8tV4deH0WSBVqArvw9lefTJh8ADXV pXsC1cHQ+3lLLrRxLNd8GL5P0b4G6Iec0HFakD/2YWd/X0PQFtDMtC/wQ4G+/e9B9PQ35V jgxXavpUe5fFPlP036K1YXoJ66/7Vys= Received: by mail-yw1-f176.google.com with SMTP id 00721157ae682-561b7729a12so35138187b3.1 for ; Thu, 08 Jun 2023 18:32:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686274372; x=1688866372; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=dPoyGlkGpegFUTFofVDIHWRBddk7VxC7U3Cl7g+dttQ=; b=E4DRQK86HkB1M0oL3CGdKUO2AI9NqeHpyBpHLli2aF+Pqlit3JP/3FLe7keJkIgcP9 Ui3kDjUhzLuUpOKUr/BBsh3paM313vr/kZTbKcMrzy1rnCosbjtatNixhCUekN/A7pkf A1U0Xx3rYpDhvjGoSHww7Px2zZjjdooHl41F3udSS11WEeWkhqn3r8hfZs7GDrCg2BMs cW8rdWF1rLhD4DHvb+mCyaR83x60p+qcJdlOzarzjsn+EOUWDt8OE4eCU+8SOfMHlxy4 XQHaWxY2xWWX/2GxHjET5xLBgwJ5RGHZxlXYqQkMUK5vHAdHkIfUkQT70qyxIHs4Ypp2 ur1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686274372; x=1688866372; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=dPoyGlkGpegFUTFofVDIHWRBddk7VxC7U3Cl7g+dttQ=; b=PDYEqr87WszKpdKk68a3Po4ML7xcTlMI7ZK3/Od63bFxdhfa7NJAt4F7TBNDNxtSqK pBuRpPwH5/DaXRa1UGik/hJwH7IxixOTPWal0mGQkeUvbQAQ1wabvg24VSwMAfGkh6Qb xknup/HwIiTe9DC4mCqo6bk6TcZxIxsy1lk51ISJGoxwgefOhUirn1x24oQ77qMsUt7+ AB9msv9lGJQZMeZQcAS09Q5rwXDN2V7Npp63iu/5K7nG1uepqSJAmcbG3LyYhxIOubND vTtd/tY6iR/mDCyRCtYicah+t6mOex7+0jGD778BB+zyWbePt5c++qzwweF2uez/QV9c KE6Q== X-Gm-Message-State: AC+VfDx/M4JrFPa1wOcAknVRnQbia7ewNpL1Kbo5qlz86WZYTg3raHQc RrEDFt/Y7ruJhZIV5f5Ph1xdYQ== X-Google-Smtp-Source: ACHHUZ6C2iTOD9MuiuvdKguC1n6gRPJbYpY/VQ3aTwN0L3f5aJC3JH3HR0E7O5S3i52MFj8yhumCLw== X-Received: by 2002:a81:4f4c:0:b0:561:1cb6:f3d6 with SMTP id d73-20020a814f4c000000b005611cb6f3d6mr549240ywb.0.1686274371904; Thu, 08 Jun 2023 18:32:51 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id u195-20020a0debcc000000b00565c29cf592sm309752ywe.10.2023.06.08.18.32.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:32:51 -0700 (PDT) Date: Thu, 8 Jun 2023 18:32:47 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 19/32] mm/mremap: retry if either pte_offset_map_*lock() fails In-Reply-To: Message-ID: <65e5e84a-f04-947-23f2-b97d3462e1e@google.com> References: MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 147F32001A X-Stat-Signature: d91ni8bnhqidi94iuk7fatno7sbq6qco X-HE-Tag: 1686274372-303270 X-HE-Meta: U2FsdGVkX19gGqFopOOlOTSdQppBOl13MC57YXiW0r7tOWhOFMXBvleLEjouMbToBW1clV69gU6pehYkTYcvCWqeKOu8rATpgEpq0LZ9NBLjS7Sa6V+SD4SUZNFwX2Zn/X0kQooNqZbrGHR46f/QXXrJSOLVoDNV7c+h4HR9v4USW5RBCxiMlM7cikaskwSyCwxp9bzo0OFshpwKsopu3Qdobru2hKIZe7pcLqDzdqm8zSU90puPHfyaI7+Mck/GoObgotnV9VcPP2oYbMCUGXdYSP346OP1AQRwGvXQ/6bN91MLkC1n9BvCGRCWmhqdRaB0VxCWAbL7mT52p8VCzcfg3oLofiofIpAodL/Ob7DG839fI7ng+S5SPKmlenAkgB+IBKD/JJv6w/LQ8uEZK7CL5+YnPhy9b5h124EViGpjA6QzSxu4cSsQ5idwn5GXsx5JyX/iVZkPf8eiZH59PNwwIvPIxzvIo25Rc5POdWijUww25xcRRlKfPXZwWHB52EWq4mUO71L/ZIpWm2yOscJQLK0U/PZ5vwtuHHf65wt8LuLClx+wkiQMpK0jM9USfJe9AUmenuB7pbW7XqcJ34NzputykLTCGPYrzEgJqYaddcRGySrUL2NCw1CTADYL6OrxKq2130OnXiKDrbBTi85SzFffoyIFHhOjugFtlrLEMkDhDRR4JHNYZjYEH7mZGdWzG+aGY621OeOzdJxp6IoeUlNBUC0B2BOq5/jDDfeTETOmjhBwtyKYo8fKazfDtaLMSgX4gdWTCpT8jSa2CipvYyvajI0kVRivEs0+M+RRpKeYM1kSARnQiMAuYz/s3pBSbvoF2ZYQNjYnov6QOPWh/2CBrQZVAAxF4krXnxvn6vuTyZLQF1rENc/FX3AlzNP9Fs+Gl3lRxrYXRk2gBtzdk5Jrr9In/YEwOY4cizkHxm5VK+NkDctebFD0ck9TKvg7BZ2WQ2SY04yLq7G 6S15iPdt IlS/FlFamTe7uCSOFj9ZeK52eszRq/avU5WBQqCxMitNPAIvs58zQNXfbt5kOUW0/5opBg+MGcbeGppJtOmPxBH6egOn/CsXU5zHJ7irokhOAvPVXBN8NNPXF0SKisADrdWPmiWzm1TrAbWLm1Tu7cPbucabaaTwzIt2Io7bTH3Mp9nOkMcIihADdAPW9DU0aIRrcIMU5OTcU7B9ucp8JlC+pnybCGsXxTdZ2M9HayjetarXLEjfQXSZhtCZpVU2t0r4BPLDeO7xaltMyIEy3DhwQzCKYcwzS+PpGIihSOJoyVtr0fyBsc70lK4ST0xOq7C6BE/giW4LGDi3A31UVbWSZbCzH0Z40VkpKpRi6LEkEcnGirVWc7x7XDK0LV8++r2QmRH29n/CLrqf69skBWVXS4GedB+9pNPlAUal60zwWx3oTLOxbXkbuAg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: move_ptes() return -EAGAIN if pte_offset_map_lock() of old fails, or if pte_offset_map_nolock() of new fails: move_page_tables() retry if so. But that does need a pmd_none() check inside, to stop endless loop when huge shmem is truncated (thank you to syzbot); and move_huge_pmd() must tolerate that a page table might have been allocated there just before (of course it would be more satisfying to remove the empty page table, but this is not a path worth optimizing). Signed-off-by: Hugh Dickins --- mm/huge_memory.c | 5 +++-- mm/mremap.c | 28 ++++++++++++++++++++-------- 2 files changed, 23 insertions(+), 10 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 624671aaa60d..d4bd5fa7c823 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1760,9 +1760,10 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr, /* * The destination pmd shouldn't be established, free_pgtables() - * should have release it. + * should have released it; but move_page_tables() might have already + * inserted a page table, if racing against shmem/file collapse. */ - if (WARN_ON(!pmd_none(*new_pmd))) { + if (!pmd_none(*new_pmd)) { VM_BUG_ON(pmd_trans_huge(*new_pmd)); return false; } diff --git a/mm/mremap.c b/mm/mremap.c index b11ce6c92099..1fc47b4f38d7 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -133,7 +133,7 @@ static pte_t move_soft_dirty_pte(pte_t pte) return pte; } -static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, +static int move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, unsigned long old_addr, unsigned long old_end, struct vm_area_struct *new_vma, pmd_t *new_pmd, unsigned long new_addr, bool need_rmap_locks) @@ -143,6 +143,7 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, spinlock_t *old_ptl, *new_ptl; bool force_flush = false; unsigned long len = old_end - old_addr; + int err = 0; /* * When need_rmap_locks is true, we take the i_mmap_rwsem and anon_vma @@ -170,8 +171,16 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, * pte locks because exclusive mmap_lock prevents deadlock. */ old_pte = pte_offset_map_lock(mm, old_pmd, old_addr, &old_ptl); - new_pte = pte_offset_map(new_pmd, new_addr); - new_ptl = pte_lockptr(mm, new_pmd); + if (!old_pte) { + err = -EAGAIN; + goto out; + } + new_pte = pte_offset_map_nolock(mm, new_pmd, new_addr, &new_ptl); + if (!new_pte) { + pte_unmap_unlock(old_pte, old_ptl); + err = -EAGAIN; + goto out; + } if (new_ptl != old_ptl) spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); flush_tlb_batched_pending(vma->vm_mm); @@ -208,8 +217,10 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, spin_unlock(new_ptl); pte_unmap(new_pte - 1); pte_unmap_unlock(old_pte - 1, old_ptl); +out: if (need_rmap_locks) drop_rmap_locks(vma); + return err; } #ifndef arch_supports_page_table_move @@ -537,6 +548,7 @@ unsigned long move_page_tables(struct vm_area_struct *vma, new_pmd = alloc_new_pmd(vma->vm_mm, vma, new_addr); if (!new_pmd) break; +again: if (is_swap_pmd(*old_pmd) || pmd_trans_huge(*old_pmd) || pmd_devmap(*old_pmd)) { if (extent == HPAGE_PMD_SIZE && @@ -544,8 +556,6 @@ unsigned long move_page_tables(struct vm_area_struct *vma, old_pmd, new_pmd, need_rmap_locks)) continue; split_huge_pmd(vma, old_pmd, old_addr); - if (pmd_trans_unstable(old_pmd)) - continue; } else if (IS_ENABLED(CONFIG_HAVE_MOVE_PMD) && extent == PMD_SIZE) { /* @@ -556,11 +566,13 @@ unsigned long move_page_tables(struct vm_area_struct *vma, old_pmd, new_pmd, true)) continue; } - + if (pmd_none(*old_pmd)) + continue; if (pte_alloc(new_vma->vm_mm, new_pmd)) break; - move_ptes(vma, old_pmd, old_addr, old_addr + extent, new_vma, - new_pmd, new_addr, need_rmap_locks); + if (move_ptes(vma, old_pmd, old_addr, old_addr + extent, + new_vma, new_pmd, new_addr, need_rmap_locks) < 0) + goto again; } mmu_notifier_invalidate_range_end(&range); From patchwork Fri Jun 9 01:34:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273084 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51C64C7EE29 for ; Fri, 9 Jun 2023 01:34:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E498E8E0003; Thu, 8 Jun 2023 21:34:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DFA278E0001; Thu, 8 Jun 2023 21:34:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CC28A8E0003; Thu, 8 Jun 2023 21:34:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id BFAA68E0001 for ; Thu, 8 Jun 2023 21:34:11 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 494F6803AF for ; Fri, 9 Jun 2023 01:34:11 +0000 (UTC) X-FDA: 80881488702.21.37ACF34 Received: from mail-yw1-f172.google.com (mail-yw1-f172.google.com [209.85.128.172]) by imf30.hostedemail.com (Postfix) with ESMTP id 6395E80004 for ; Fri, 9 Jun 2023 01:34:08 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=Oi7q9zQX; spf=pass (imf30.hostedemail.com: domain of hughd@google.com designates 209.85.128.172 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686274448; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wdvmzniCtM+nQioqQdrwu6IzqdoeozaKC+WtG0d7jzc=; b=CWBurM42/6GZon9eDX+WV+jKj0cBPriCuWp6JjmQCuc9I4w3v38BtAQME80PpZqRidLTW7 PB3HHTLpUpVKhTmlpFH2du2l8TGLggMKCpoDuQTMD8nk1W2zGVCmopf+exmVvM7F71Nn9+ YStWaWyIn+qOMY2EUxmSULlU1qLvtBA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686274448; a=rsa-sha256; cv=none; b=ldeHSPgNJPfoL1Xq8WPp6iXuiGVnwIvPgmBSITGOPJsPjgevZumV/WPZ3oToQJVyUGrgL4 yUw8C6DZektprf7XyH2eMnOdm/XWhMAkse3OzTjgzEFJyUm5UoplaX40ZYoE1G3hbNfcgX /CRH4hUiQn+NM5NQ2AHz+4C3vr8zu9U= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=Oi7q9zQX; spf=pass (imf30.hostedemail.com: domain of hughd@google.com designates 209.85.128.172 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f172.google.com with SMTP id 00721157ae682-565e6beb7aaso11623917b3.2 for ; Thu, 08 Jun 2023 18:34:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686274447; x=1688866447; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=wdvmzniCtM+nQioqQdrwu6IzqdoeozaKC+WtG0d7jzc=; b=Oi7q9zQXAnHzVKo/FhCPs8tzz9nacAj7GeJ1aGGBUrWEYASbA/COShPsGhE7TSY1ih bxVmRvKdrmFR5jXzZRTgemTvq61YU0XHRU70b1d3QdbaFwgY+kX5E2pxmuHyonISfka/ 2mIjcxl1BRTJLCdi9tDPNj4z8xlbHAAh2ERkE6DkidseXIGdbxnscT3F7ARVo5KkhPVF m5RsG7gDmqBonOEOuZYs+SmyuTOVvsJthOrxfrS4j3QsaT2MI33gyJZxrSrSJZSC8FPD R5ZnuYh4R1BXHnQy+G2iJxg5x1Uo6TM759fPwJnBzBOJAwDjI9gtM3K7QZ52HU2U68va CGEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686274447; x=1688866447; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wdvmzniCtM+nQioqQdrwu6IzqdoeozaKC+WtG0d7jzc=; b=kHXoRF5mHLigMV3pzHs+F+ZJJQwXXSTe8w6t3lWvbX1k8IEFMyDPc8cKc9KgeODGCN ZczHLXEUrZFIHG3a2I/rN4zHkNOtCaahX7zy2fJNPj0lxfb++fWUZtPN770U9jciTnsk w1ffcspi1hmCwwup6o/Ak2nmhBVFrRARbPA4KhBBDoIWkFBCivIZQrMn7wnv4ZwlTsNf 68TSR133173cK0DTno9bzpk1WaHFNOfgMz5+nEPSZWVcd4Kh2q2d6/RrohWoWXDRcjA+ Q4jaymgaHfn2TnEFzBPNOGRKYUjxBl6bz67v9CaugxGcnsY2iKpvNJeevFEAm4dKNV9B 87yg== X-Gm-Message-State: AC+VfDzaGmmhcYinzeJlHt7GyNPqS8K30cU4JASvzBhAOpaoeRDWawth rwbR6ZnrqEND8FkNCayHneGSUg== X-Google-Smtp-Source: ACHHUZ5Q/YHvHVewBNkmQMz1HqeMOeXDnHdCMp674SssIxi8mNPsiGfn7zCsWmM3qzcif+cYyekxpQ== X-Received: by 2002:a0d:cb50:0:b0:54d:ea34:c31 with SMTP id n77-20020a0dcb50000000b0054dea340c31mr1301109ywd.29.1686274447361; Thu, 08 Jun 2023 18:34:07 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id q67-20020a818046000000b0054f83731ad2sm314580ywf.0.2023.06.08.18.34.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:34:06 -0700 (PDT) Date: Thu, 8 Jun 2023 18:34:03 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 20/32] mm/madvise: clean up pte_offset_map_lock() scans In-Reply-To: Message-ID: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 6395E80004 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: asfyxd7ce3nnwjqy6bu47cyguudjhyhr X-HE-Tag: 1686274448-215513 X-HE-Meta: U2FsdGVkX18ijAGWz9KVmVYZHpODzz91VlSBkD7j/ijsnsl9Igb63vRKWGI4yGOGSiXfzOabZSz1DZHGS2N09gJa6bbWt2RpsAVW5qe4JmMTSppRJR6+3oufJg1OoRGCRylnyS1K96uNHpFMpWx1O2nlJdGMYU0+in0zpaC2wb3LUeOzZofct0e7bCw7IoHab6w7Wj6xWnCJnEHxmgqRkaWwVMxMUMuUbfTgNx4s9MTFy+sEi4WxCLK8I6vSSYaCu6Xo/58B2vZiFWCJMcvvSwoXoeFq6EnnjU/Cgwi+b5AbCfbD4iNCKeH6tBVvwbZF9fmnDPtRGAzcvWYrz0tDaIfrx6+8i2EjM3chgwmixQCAyj27hS44fvAsW+aYHPzKTGi8/WMVd5OOS/kI6vK9g6ViMNXocbApaMF9ErC6vZG6CDLdj0r+X+ECbTL8By3ojEdmqZg/nri96tN6XxW0JBln+7l8tx2GOyUUhDJTziGUIuj6Dn8YFhZ51t5lh/aV2n2Apgy+c27WqROLXqftRVuctFdJ9PLgEkyfiSzZAt1HxAs/zpFc08bgn5jZsT8t/WEXDLURNhJ/kreWIe1qtxH7uVoYS/cLN4NZ/KN7OHnJJG/YK8UkG/S5KaQEQvuMU1v28iM8oA5nYT9/nXkCgBtJz9bBN/B6NwwTs9WFfZKueVtqpuN2KDaw5R/YrKD0cdzabi6QSxR35CFXJio291x+z0DW5AJYQnauGeIgkUwcEs0UJ9SKd7DqWp+OqJByAbtJ2bohf8uTg3huh4fZvMpyGIZnY7S6nYfYQq4s/bwqaTvw/9K2Dqb7XXSinLPsf+v4BK4tXeyCIdwTQzB2fW7jL7l2de3sFE1qPP8C9KaFcWHNsqdPTCPiFzeLKZzacEJDosfaszHP/R6JTXh9t4aNNlW1fQYfzdXQXeByz5a2I12IQZxdJfj7PA/TjcCWd26NtSv9T2b/eLX7dT5 dye3BLE1 zR8VNpa+t5L9oayqnC3jJPNi83Izd6tlUQetk3DD5eBVTHvuTI/RaPVIrbhnG1a9FkEP7LnuFJxXoeoUzSK/KPkRcEVuv/pzmLEjvA0IFxMe0CrwTtgKQdQHYvzYbBdXJI0tKVnZUf9Y1BXBti/ta9l+uMRbXua61GQXmLmZ45Ydr0SxrVSwEawCunMyY23FK6tA2KaStIaX1yOfG4j2SELNn74hwaC/IsC+jWMhV9AbjDdp3HXJa5qUUgWRKXmap6K7tzklyrv3uuadtOtukZcOUqwpbZzWlAiVggoyTDu9274SXDF/Un9NlrHglSVLQW2fglaTiudAvU8Jo/94RhnDUDQ9TPkUxpLalTAWb9OQWwE4ynn9v/EED0a6dFvpjzyQt3/DRY06bm96ZOP+4HTxYufrSAGdCYYWtBRZk4ifF3Whbfohog7uYIQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Came here to make madvise's several pte_offset_map_lock() scans advance to next extent on failure, and remove superfluous pmd_trans_unstable() and pmd_none_or_trans_huge_or_clear_bad() calls. But also did some nearby cleanup. swapin_walk_pmd_entry(): don't name an address "index"; don't drop the lock after every pte, only when calling out to read_swap_cache_async(). madvise_cold_or_pageout_pte_range() and madvise_free_pte_range(): prefer "start_pte" for pointer, orig_pte usually denotes a saved pte value; leave lazy MMU mode before unlocking; merge the success and failure paths after split_folio(). Signed-off-by: Hugh Dickins --- mm/madvise.c | 122 ++++++++++++++++++++++++++++----------------------- 1 file changed, 68 insertions(+), 54 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index b5ffbaf616f5..0af64c4a8f82 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -188,37 +188,43 @@ static int madvise_update_vma(struct vm_area_struct *vma, #ifdef CONFIG_SWAP static int swapin_walk_pmd_entry(pmd_t *pmd, unsigned long start, - unsigned long end, struct mm_walk *walk) + unsigned long end, struct mm_walk *walk) { struct vm_area_struct *vma = walk->private; - unsigned long index; struct swap_iocb *splug = NULL; + pte_t *ptep = NULL; + spinlock_t *ptl; + unsigned long addr; - if (pmd_none_or_trans_huge_or_clear_bad(pmd)) - return 0; - - for (index = start; index != end; index += PAGE_SIZE) { + for (addr = start; addr < end; addr += PAGE_SIZE) { pte_t pte; swp_entry_t entry; struct page *page; - spinlock_t *ptl; - pte_t *ptep; - ptep = pte_offset_map_lock(vma->vm_mm, pmd, index, &ptl); + if (!ptep++) { + ptep = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + if (!ptep) + break; + } + pte = *ptep; - pte_unmap_unlock(ptep, ptl); - if (!is_swap_pte(pte)) continue; entry = pte_to_swp_entry(pte); if (unlikely(non_swap_entry(entry))) continue; + pte_unmap_unlock(ptep, ptl); + ptep = NULL; + page = read_swap_cache_async(entry, GFP_HIGHUSER_MOVABLE, - vma, index, false, &splug); + vma, addr, false, &splug); if (page) put_page(page); } + + if (ptep) + pte_unmap_unlock(ptep, ptl); swap_read_unplug(splug); cond_resched(); @@ -340,7 +346,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, bool pageout = private->pageout; struct mm_struct *mm = tlb->mm; struct vm_area_struct *vma = walk->vma; - pte_t *orig_pte, *pte, ptent; + pte_t *start_pte, *pte, ptent; spinlock_t *ptl; struct folio *folio = NULL; LIST_HEAD(folio_list); @@ -422,11 +428,11 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, } regular_folio: - if (pmd_trans_unstable(pmd)) - return 0; #endif tlb_change_page_size(tlb, PAGE_SIZE); - orig_pte = pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + start_pte = pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + if (!start_pte) + return 0; flush_tlb_batched_pending(mm); arch_enter_lazy_mmu_mode(); for (; addr < end; pte++, addr += PAGE_SIZE) { @@ -447,25 +453,28 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, * are sure it's worth. Split it if we are only owner. */ if (folio_test_large(folio)) { + int err; + if (folio_mapcount(folio) != 1) break; if (pageout_anon_only_filter && !folio_test_anon(folio)) break; + if (!folio_trylock(folio)) + break; folio_get(folio); - if (!folio_trylock(folio)) { - folio_put(folio); - break; - } - pte_unmap_unlock(orig_pte, ptl); - if (split_folio(folio)) { - folio_unlock(folio); - folio_put(folio); - orig_pte = pte_offset_map_lock(mm, pmd, addr, &ptl); - break; - } + arch_leave_lazy_mmu_mode(); + pte_unmap_unlock(start_pte, ptl); + start_pte = NULL; + err = split_folio(folio); folio_unlock(folio); folio_put(folio); - orig_pte = pte = pte_offset_map_lock(mm, pmd, addr, &ptl); + if (err) + break; + start_pte = pte = + pte_offset_map_lock(mm, pmd, addr, &ptl); + if (!start_pte) + break; + arch_enter_lazy_mmu_mode(); pte--; addr -= PAGE_SIZE; continue; @@ -510,8 +519,10 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, folio_deactivate(folio); } - arch_leave_lazy_mmu_mode(); - pte_unmap_unlock(orig_pte, ptl); + if (start_pte) { + arch_leave_lazy_mmu_mode(); + pte_unmap_unlock(start_pte, ptl); + } if (pageout) reclaim_pages(&folio_list); cond_resched(); @@ -612,7 +623,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, struct mm_struct *mm = tlb->mm; struct vm_area_struct *vma = walk->vma; spinlock_t *ptl; - pte_t *orig_pte, *pte, ptent; + pte_t *start_pte, *pte, ptent; struct folio *folio; int nr_swap = 0; unsigned long next; @@ -620,13 +631,12 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, next = pmd_addr_end(addr, end); if (pmd_trans_huge(*pmd)) if (madvise_free_huge_pmd(tlb, vma, pmd, addr, next)) - goto next; - - if (pmd_trans_unstable(pmd)) - return 0; + return 0; tlb_change_page_size(tlb, PAGE_SIZE); - orig_pte = pte = pte_offset_map_lock(mm, pmd, addr, &ptl); + start_pte = pte = pte_offset_map_lock(mm, pmd, addr, &ptl); + if (!start_pte) + return 0; flush_tlb_batched_pending(mm); arch_enter_lazy_mmu_mode(); for (; addr != end; pte++, addr += PAGE_SIZE) { @@ -664,23 +674,26 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, * deactivate all pages. */ if (folio_test_large(folio)) { + int err; + if (folio_mapcount(folio) != 1) - goto out; + break; + if (!folio_trylock(folio)) + break; folio_get(folio); - if (!folio_trylock(folio)) { - folio_put(folio); - goto out; - } - pte_unmap_unlock(orig_pte, ptl); - if (split_folio(folio)) { - folio_unlock(folio); - folio_put(folio); - orig_pte = pte_offset_map_lock(mm, pmd, addr, &ptl); - goto out; - } + arch_leave_lazy_mmu_mode(); + pte_unmap_unlock(start_pte, ptl); + start_pte = NULL; + err = split_folio(folio); folio_unlock(folio); folio_put(folio); - orig_pte = pte = pte_offset_map_lock(mm, pmd, addr, &ptl); + if (err) + break; + start_pte = pte = + pte_offset_map_lock(mm, pmd, addr, &ptl); + if (!start_pte) + break; + arch_enter_lazy_mmu_mode(); pte--; addr -= PAGE_SIZE; continue; @@ -725,17 +738,18 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, } folio_mark_lazyfree(folio); } -out: + if (nr_swap) { if (current->mm == mm) sync_mm_rss(mm); - add_mm_counter(mm, MM_SWAPENTS, nr_swap); } - arch_leave_lazy_mmu_mode(); - pte_unmap_unlock(orig_pte, ptl); + if (start_pte) { + arch_leave_lazy_mmu_mode(); + pte_unmap_unlock(start_pte, ptl); + } cond_resched(); -next: + return 0; } From patchwork Fri Jun 9 01:35:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273085 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 777C8C7EE29 for ; Fri, 9 Jun 2023 01:35:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1780F8E0002; Thu, 8 Jun 2023 21:35:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1282E8E0001; Thu, 8 Jun 2023 21:35:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 017928E0002; Thu, 8 Jun 2023 21:35:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id E840E8E0001 for ; Thu, 8 Jun 2023 21:35:21 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id B9DF21A0369 for ; Fri, 9 Jun 2023 01:35:21 +0000 (UTC) X-FDA: 80881491642.15.DC982BC Received: from mail-yw1-f179.google.com (mail-yw1-f179.google.com [209.85.128.179]) by imf10.hostedemail.com (Postfix) with ESMTP id EC323C0002 for ; Fri, 9 Jun 2023 01:35:19 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=Sak7mjF3; spf=pass (imf10.hostedemail.com: domain of hughd@google.com designates 209.85.128.179 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686274520; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hzsL+M6cdepZgQw4hIUfWIci6E11MaKNkaWhV1yArkY=; b=y7cOZfrmEC9DGwg6pOMEo3UG8woSL680vWv27qldTMi1x0skHaKS1wMmoHlrzcI47Gx847 ckmnCjOnHrpazhI5fAaWKmsO/7r0nZRdcUMmdqNlJmGkHKWAxjHleLqs2awmV4No43ieex 7pCmoPqfkM3aD7RXq2kcHwlKYTprfew= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686274520; a=rsa-sha256; cv=none; b=EZsNv2JdRHteui/K5nwak7cjG+aetay/pAtc4ppfiLOGsyISW5K9aKfnP06xc5FnqFv2Ci epoYJR1wSQ0K2m2Q9UZ2XAn9wgoaObW7cXjoKW1vDUcjj9f2oIiT/95XB1pXbNiSAiB4da cA1ua41+YMFH9jZlTUIatLsqfv/R5kw= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=Sak7mjF3; spf=pass (imf10.hostedemail.com: domain of hughd@google.com designates 209.85.128.179 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f179.google.com with SMTP id 00721157ae682-565cdb77b01so10931267b3.0 for ; Thu, 08 Jun 2023 18:35:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686274519; x=1688866519; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=hzsL+M6cdepZgQw4hIUfWIci6E11MaKNkaWhV1yArkY=; b=Sak7mjF34nMHTvCHP5NeKRtkES8TSscUHlyC9R0R46SCrhEJJ2Wo+n7dfWrX77uQe8 nf+QSJZI16aTp4GUl6IGu9WutySoxO4BhydtYaduPdP51lxLi8Eo7UzEJvf+9zsLlQKg 1uAWQEdqAQK63Qji2NQKBkmHFaElkPHhKBew+iyAcD1FGdj+qH73dMkXTiCWxIZUVbNy fXN+aUpQp9yRvXbnP1DZhFGyMAkNlSUaOV8BRQ3aucz5IbMfcsCH3z+WaSedYI9tzU4M sRC3DhkA8g6Vrf7EurgcGfyAeEcYsIRAt7OjP2hbNUKH6TRoKes6M9tBjy+ai0vdg3/i pdHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686274519; x=1688866519; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=hzsL+M6cdepZgQw4hIUfWIci6E11MaKNkaWhV1yArkY=; b=ggsO9jMdKWyKVSPr3RFLjlemLBebfKsnUlg0lxenOHKcv3IvFQB9kkkEyVhALZT6Fr /efWhJDUHaXFHZeBxQrvJTKKs737Fg8DXpma6uVmMGe/UQVWxqNY7vg4CjWI/T+IAByA pRQ0sEYQ96qmuUCa1bLhr22oYv7mehSRuhm87TCYS5Xq3feJtXipUj+D60mvkyB4NdzV 9beVYBqBA7kZCkBHGpcehfx1RTZp+xonNzh2NWotDKu1tiU0b9i8qoIBiSqMHF6r2PTZ 4s5DyW7q9UgtYVdgb1tHbvyNYaDXkJRG/F84UVY8EvnVlkleaOH2W/ugMJMeBHrPXBiL QL5w== X-Gm-Message-State: AC+VfDxu4WyOqBOarzRdDHzw+ikMIlrkon7yos4hb+SM8sTlpKg4tF5Q mYBq26BTyuGvO5lAb/QsKVmT0A== X-Google-Smtp-Source: ACHHUZ4AK/lYHw0bDyLVKmEksYxVFlrDP4tGf35FRUzn6cSvs6hJCxMo87mkPkKs2nriI5iLhLUP+g== X-Received: by 2002:a81:4e04:0:b0:568:ea0e:ae75 with SMTP id c4-20020a814e04000000b00568ea0eae75mr1286809ywb.45.1686274518944; Thu, 08 Jun 2023 18:35:18 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id s4-20020a81bf44000000b0054601ee157fsm283836ywk.114.2023.06.08.18.35.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:35:18 -0700 (PDT) Date: Thu, 8 Jun 2023 18:35:14 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 21/32] mm/madvise: clean up force_shm_swapin_readahead() In-Reply-To: Message-ID: <67e18875-ffb3-ec27-346-f350e07bed87@google.com> References: MIME-Version: 1.0 X-Stat-Signature: fipcg36wjdc5g9r31sgx397n1bmeedtd X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: EC323C0002 X-Rspam-User: X-HE-Tag: 1686274519-740381 X-HE-Meta: U2FsdGVkX1/lex79Zmlco44sbX/Vb8a9eTWBSCN3XKJ7l+ML3gVASyS/ITY48+tPGm8zzxPZr7HiQBp9oXzkCm/yKWPA1Bd8EmAT5hLWuC4N5LkxbaCzrNtThqssJNRhbM6H1Ip9k3EUAHpjVjk9LtHvO4Y77upK4Wl8yOpMdF3re+W8r1c09tVPSlkRnVHhB8x9Y4F9rLm8I40cB1uMgdZYN5kFXWClFuAdsqzFbAchKmSKAl/CDPdD45YLpxjgierCIZa0S7zYXBn2KW90yPVq0aePn13SzEvzo8lRlgMmqnJiVsWBxs13YjMT5DQ+JInvJdp27J6m5LSdU4UTw5KndNd2u4l5MYjCmrVNQm9CiCP30kPO5tkokfoRGbVomSf731GBHJavRToSb5cWzg1/b9qkYQs9jFc8BPKLcyegFrLPCvxIUZjnBaqp4W5FvPh0MDSVGUqLxhGBgVUXzPZkPjxO6eRypcmraKPHdvEBfWsL7Ki4IbMY5xWiD07/32iZu1rxBFohlkxn0bhXOs0WNXMPULfUY4CdwCdgixyxz6eMR4prTOfQ4ox5z4pZ+R1R6sN2n9VEuL/heovwvkO7LpBiJHjfHF1M11+9pPXztcbcH5Qaqm1Yk4JXZSH69PGUEkxNEWSX9uhy3OqloBFpcf+If0fpyLqbK1k4SKzq3hHbwUuJ5MYu1xhyTOSHvdMdIhzsXCetbr6iO1bkrL0ybdGmELAA5kPik7ru5elGCcRYObWAaURplHvRRzGK57v9324j1Ap/rVYVTkInXQZ0RQ9g1pHV/eFQjE3Gjx74DDETZoj2AqitRQCbBrraXuhQs9GMBRcXVnPjpsWl6c22Wasc1dMmQ2jzNhoy/qV4m3Pdb8hwKFIzlFJXH7TpB2DaTOLjdxc0jMWr0/62GnO5EzG57S92+u5LmqL3cefMBhIqR+OKtDK5moAmjKBfky8r0NDi58VgPSFmdiU ypog0BGO /bB3Ll2HP0G1l3mhzNE0aH4QrXPWAMh8OMbockuZMpNA6Yy+7y06pfCzj70u1nC+mY9Qc6xVmbbfroEp1pl6CC9kf7QPk1ij+muWH93WUawWXyPdu9c9H5r07Ey6Bj1bs9SRV7NUeK7wsdQHbO7BjohCQOlirvxWOqCjIfvUdCF90eO6+MC0DuH4FGuVHBTXtHRXMsRvEqLqxY+WdnVwhmgtyslfJbvhGDnDORChIQC/qlE6W7iI/HC1Y1jmVDoKrMkkXwSHIt3ueE3fIh8fF4Z1zU2OsFsk11HmIkKWdWXgZGHCXGgsgqyRreSgIf+wlfD+h4Jy/T/f5kiPBd7q1tBHq+U0MzoxZncuhoCft35OOQeKE5Fp3kQy8KZSTQDAs8bnjdHEUOKzsayCgK5AviS0m44rDYJ/Cjk0QSRDpnEAD7fZl+HaSCNN11g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Some nearby MADV_WILLNEED cleanup unrelated to pte_offset_map_lock(). shmem_swapin_range() is a better name than force_shm_swapin_readahead(). Fix unimportant off-by-one on end_index. Call the swp_entry_t "entry" rather than "swap": either is okay, but entry is the name used elsewhere in mm/madvise.c. Do not assume GFP_HIGHUSER_MOVABLE: that's right for anon swap, but shmem should take gfp from mapping. Pass the actual vma and address to read_swap_cache_async(), in case a NUMA mempolicy applies. lru_add_drain() at outer level, like madvise_willneed()'s other branch. Signed-off-by: Hugh Dickins --- mm/madvise.c | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index 0af64c4a8f82..9b3c9610052f 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -235,30 +235,34 @@ static const struct mm_walk_ops swapin_walk_ops = { .pmd_entry = swapin_walk_pmd_entry, }; -static void force_shm_swapin_readahead(struct vm_area_struct *vma, +static void shmem_swapin_range(struct vm_area_struct *vma, unsigned long start, unsigned long end, struct address_space *mapping) { XA_STATE(xas, &mapping->i_pages, linear_page_index(vma, start)); - pgoff_t end_index = linear_page_index(vma, end + PAGE_SIZE - 1); + pgoff_t end_index = linear_page_index(vma, end) - 1; struct page *page; struct swap_iocb *splug = NULL; rcu_read_lock(); xas_for_each(&xas, page, end_index) { - swp_entry_t swap; + unsigned long addr; + swp_entry_t entry; if (!xa_is_value(page)) continue; - swap = radix_to_swp_entry(page); + entry = radix_to_swp_entry(page); /* There might be swapin error entries in shmem mapping. */ - if (non_swap_entry(swap)) + if (non_swap_entry(entry)) continue; + + addr = vma->vm_start + + ((xas.xa_index - vma->vm_pgoff) << PAGE_SHIFT); xas_pause(&xas); rcu_read_unlock(); - page = read_swap_cache_async(swap, GFP_HIGHUSER_MOVABLE, - NULL, 0, false, &splug); + page = read_swap_cache_async(entry, mapping_gfp_mask(mapping), + vma, addr, false, &splug); if (page) put_page(page); @@ -266,8 +270,6 @@ static void force_shm_swapin_readahead(struct vm_area_struct *vma, } rcu_read_unlock(); swap_read_unplug(splug); - - lru_add_drain(); /* Push any new pages onto the LRU now */ } #endif /* CONFIG_SWAP */ @@ -291,8 +293,8 @@ static long madvise_willneed(struct vm_area_struct *vma, } if (shmem_mapping(file->f_mapping)) { - force_shm_swapin_readahead(vma, start, end, - file->f_mapping); + shmem_swapin_range(vma, start, end, file->f_mapping); + lru_add_drain(); /* Push any new pages onto the LRU now */ return 0; } #else From patchwork Fri Jun 9 01:36:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273087 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F7E8C7EE25 for ; Fri, 9 Jun 2023 01:36:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 98E168E0002; Thu, 8 Jun 2023 21:36:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9175F8E0001; Thu, 8 Jun 2023 21:36:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 76ACB8E0002; Thu, 8 Jun 2023 21:36:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 3BC5E8E0001 for ; Thu, 8 Jun 2023 21:36:18 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 07EED1A039B for ; Fri, 9 Jun 2023 01:36:18 +0000 (UTC) X-FDA: 80881494036.12.38AA773 Received: from mail-yw1-f175.google.com (mail-yw1-f175.google.com [209.85.128.175]) by imf07.hostedemail.com (Postfix) with ESMTP id 431D24000F for ; Fri, 9 Jun 2023 01:36:16 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=a5ewmipk; spf=pass (imf07.hostedemail.com: domain of hughd@google.com designates 209.85.128.175 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686274576; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7TjV4RyPitx7LIAI3Pj7MSABK+uluBQcp84mqUzVC6k=; b=RJ9/F+PDxUmSFmWzJzUNLxnWdLb0ZzHqMAGCJWOu3j9e4+4f+xKIVVrZhvqfar0V0Jh3jQ u/BpMpQN4WbDXqyiOc8wzapeJbgyd5xlXKWAeS3pA5+68wMyXdX2cm1aFxEoyKkkufUxM1 cp2UV+6fck9zZwgdpomsQZuXf1+VuYc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686274576; a=rsa-sha256; cv=none; b=DotJIhhMBso+MFg+VIqK1icrrX/APjCV9wB8+nGtYYoH6CABuKAYZFG3zIQbeGy4lC7Q+b 9yiM68TmrQT6hO7ga88+hj8JXREnXQ5YX65Kz6cvX0CUZPYMzgCyJ0/hvi+vOhe5MvpMGT ZpsP9PcOSoLw+osd199LW8t9yrT5k28= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=a5ewmipk; spf=pass (imf07.hostedemail.com: domain of hughd@google.com designates 209.85.128.175 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f175.google.com with SMTP id 00721157ae682-565cfe4ece7so11390117b3.2 for ; Thu, 08 Jun 2023 18:36:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686274575; x=1688866575; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=7TjV4RyPitx7LIAI3Pj7MSABK+uluBQcp84mqUzVC6k=; b=a5ewmipktwitFlsPba8meBOEdOaTniMgfwOwR66LxzGfjmzDH4IFIjppt+zCAFlN0g 4qVFbiljADzlv99p9uKv0I6LLrtlgpWP5C7ZN5jYvFhWguH9wpOq9f3yGPT9NIjCbt8f ccDNMfVoLAT69ZaBFF/iOjiPqCtqRwYPT0P1Yb6UZGbQci3A4kQCZ/DTEhdlS+lQWdYY KzmqHWulcwNNrAp2wlegE+sh4YyxODsI8618mTIxeZ3NAKuS71wgysjyvDWVHPesaDpk PafA85IWdjWaVeXqTpvoM+ur2RY2SxdVNDgFZ9WN6yrvvp/2wYtNHHpsHPpA7aoAQsWI /+rA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686274575; x=1688866575; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7TjV4RyPitx7LIAI3Pj7MSABK+uluBQcp84mqUzVC6k=; b=GumVFBAP5enzKWsn8DegwVBm4OQeq2IUNz50+ODhtlerLCaYCrduNbI45Y2EAIv54s yCuVhHKg5eQ65/U1HFKbShIJavObEWXaN58XKOmpy+FYjOOQr6C2bphCMDqRpq88vt2b 13fZ4+l6mzA8mBNumJYr+qBelrCUmm6pgW/ogc+9SgQzlwye54tsVEep4/w8lqmHJ7p0 fYNf/ghtUU3Btbl5cErB7VANytXIEJGVT4grJFtoTHRzToP9BzUNZDVrsO/TfDt0ijr+ J/1OYGfp5on9zKIuWARHI6vmtjj1RPgk1edx91YU8XE97lSYhymRUPeyBvVV3h6i8BdC y57w== X-Gm-Message-State: AC+VfDxEfZdeQtCxTWJwXGWqOOMDN/BMhkmaF3GSIDXTh0s0B68prrcE zWnzXE0ygceja2NypSVCsPtJVg== X-Google-Smtp-Source: ACHHUZ41tpEmUAG3j7GY9w97TvrJQ50NkLe4OKcO0eJ30STZxTamEDNBY6XFXEyrE/Y12cF7SVrDgA== X-Received: by 2002:a81:a0c9:0:b0:568:a870:314f with SMTP id x192-20020a81a0c9000000b00568a870314fmr1280229ywg.30.1686274575219; Thu, 08 Jun 2023 18:36:15 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id d191-20020a0ddbc8000000b00569fdf7f58bsm293760ywe.66.2023.06.08.18.36.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:36:14 -0700 (PDT) Date: Thu, 8 Jun 2023 18:36:11 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 22/32] mm/swapoff: allow pte_offset_map[_lock]() to fail In-Reply-To: Message-ID: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 431D24000F X-Rspam-User: X-Stat-Signature: jsob9jamqkzepwrufi7p1xrqiskehow1 X-Rspamd-Server: rspam03 X-HE-Tag: 1686274576-652042 X-HE-Meta: U2FsdGVkX1+MYPd41IHoliuaQ/iKz7F+zYFzLdj/SFPCRB3tIiQSkhgcb7RA0QMf1r9znZIEsoNv2G7c+K5b+yVksla/H2vUWawFpX8NsvKI17WgbElTcaXecVEnZRGAVDuAM7u6+8LwqSCQ3rG9goJVFOvVlZXml1Ew+qYINTuYf0i6VHxGmfjWpjdiOD8JPlLtAtFS8tovDyiuNi5kUE1HlO2ynanfIF6z7+Cd0Q45d7kgtZEp4k1oBIYmSrXisCjPu02rZRi1udVMLblaV3yDpoXUTcQv/9M6m2Cb7FrKgm1c4QOQWm3V+67ZyqpVKDx4aLGd07RAKc1pU7bHlnXXL+8haH5pQ49+XbcFKoSgxpD4C2LHvPz50X1yTv2qvsyJhk/uUV4heQwUMkIbY04NYF6WmfEHmwh6bblIiYSMGL6c3yYh2BGJ/aDJMtf8NFV/kIbYPUcZTCOCEAxC38lnM9T5bSuI589/Trn+yLMCdsERTc9Oq60utdFhlOBOSXFHRER7cALpII9cIn3SbuVk8A6D+I+bxD3uYkrTCniTZp3jkhOuwigFfMdhOwVaWDGX50vXSrUphUQf+CZ9apnacg28v4dLl/tPq4ui8Ua0Any/vapynqft/e1ShzSRnfut0GIox8xZjWEg0AtjZgvU/qUlqgzkaAb1DasNxBqZqQODyIdGtkcxm+2Mk3qewucJAGEwGHJ36NXBDBL9wrjlv99/QIRCuHxUcMMu3swJT0ZmqfivN+NrStHJTtj8GE42sEpp031SlF5xJG/E7q2SyPwNqOqZsb9QTZI2jvDHuNQhMJSodjZ+RR/hBkd4vAk+sH9stkkUR0ICensyOlUxi3JnaqrMj/wpXvnAuKabgL1WTTW/xIb7l8p8zCV55vpUh7Q79E+Kdsorq22FGZFXsOb6lTvDvB1EmWbpwg8hvFN7koRb9gGnjbjKzkXlO6qxaYmArxdSqTZ+DiO 9UDuXD99 BguknwRFEiQSlTIGC5zNbf3wJYpz21omKa1tfhkAvk11ePtemFTX7DAb8S4Mh+0khVuBvgyOG5Bjh6SIPl4SIvjtL5VCjYeJcqafaD0slJ5fl/wc+O9Mc6jxPmeeSp6MWITdVM++zLBiiRC/zg5i6PQItrfV1xxBMvblgf1WRXFdOUOHih0ZWtaMaA4zZBTFkGo6JDqKr/m0S4J/0WMoDCou+ppPwF3MAdoGsHbTYol/mAcUcDnlrltmUuSzPVFjvYaR4/vM30G6lvxBuSOth9X9ICFnEa+6863oGlhS8iAtftPtsY3YxQYakGz+Yy57C+1jPzcpoqMhqNrGyWhO8x5QU2lpsL8ho5e2eGYLoYkSptP9G/bzzSZaWWPzfjHwpDhgZlVVUrJyx0vnFPs9tiXYA7s2Bkl+Rvjy5daJzlU8F0jNMvt+1RQkwmQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Adjust unuse_pte() and unuse_pte_range() to allow pte_offset_map_lock() and pte_offset_map() failure; remove pmd_none_or_trans_huge_or_clear_bad() from unuse_pmd_range() now that pte_offset_map() does all that itself. Signed-off-by: Hugh Dickins --- mm/swapfile.c | 38 ++++++++++++++++++++------------------ 1 file changed, 20 insertions(+), 18 deletions(-) diff --git a/mm/swapfile.c b/mm/swapfile.c index 274bbf797480..12d204e6dae2 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1774,7 +1774,7 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, hwposioned = true; pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); - if (unlikely(!pte_same_as_swp(*pte, swp_entry_to_pte(entry)))) { + if (unlikely(!pte || !pte_same_as_swp(*pte, swp_entry_to_pte(entry)))) { ret = 0; goto out; } @@ -1827,7 +1827,8 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, set_pte_at(vma->vm_mm, addr, pte, new_pte); swap_free(entry); out: - pte_unmap_unlock(pte, ptl); + if (pte) + pte_unmap_unlock(pte, ptl); if (page != swapcache) { unlock_page(page); put_page(page); @@ -1839,17 +1840,22 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr, unsigned long end, unsigned int type) { - swp_entry_t entry; - pte_t *pte; + pte_t *pte = NULL; struct swap_info_struct *si; - int ret = 0; si = swap_info[type]; - pte = pte_offset_map(pmd, addr); do { struct folio *folio; unsigned long offset; unsigned char swp_count; + swp_entry_t entry; + int ret; + + if (!pte++) { + pte = pte_offset_map(pmd, addr); + if (!pte) + break; + } if (!is_swap_pte(*pte)) continue; @@ -1860,6 +1866,8 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, offset = swp_offset(entry); pte_unmap(pte); + pte = NULL; + folio = swap_cache_get_folio(entry, vma, addr); if (!folio) { struct page *page; @@ -1878,8 +1886,7 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, if (!folio) { swp_count = READ_ONCE(si->swap_map[offset]); if (swp_count == 0 || swp_count == SWAP_MAP_BAD) - goto try_next; - + continue; return -ENOMEM; } @@ -1889,20 +1896,17 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, if (ret < 0) { folio_unlock(folio); folio_put(folio); - goto out; + return ret; } folio_free_swap(folio); folio_unlock(folio); folio_put(folio); -try_next: - pte = pte_offset_map(pmd, addr); - } while (pte++, addr += PAGE_SIZE, addr != end); - pte_unmap(pte - 1); + } while (addr += PAGE_SIZE, addr != end); - ret = 0; -out: - return ret; + if (pte) + pte_unmap(pte); + return 0; } static inline int unuse_pmd_range(struct vm_area_struct *vma, pud_t *pud, @@ -1917,8 +1921,6 @@ static inline int unuse_pmd_range(struct vm_area_struct *vma, pud_t *pud, do { cond_resched(); next = pmd_addr_end(addr, end); - if (pmd_none_or_trans_huge_or_clear_bad(pmd)) - continue; ret = unuse_pte_range(vma, pmd, addr, next, type); if (ret) return ret; From patchwork Fri Jun 9 01:37:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273088 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB762C7EE25 for ; Fri, 9 Jun 2023 01:37:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9188C8E0002; Thu, 8 Jun 2023 21:37:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8A18A8E0001; Thu, 8 Jun 2023 21:37:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 71BCE8E0002; Thu, 8 Jun 2023 21:37:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 5CE4E8E0001 for ; Thu, 8 Jun 2023 21:37:19 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 2EC58AF090 for ; Fri, 9 Jun 2023 01:37:19 +0000 (UTC) X-FDA: 80881496598.25.38791F4 Received: from mail-yw1-f182.google.com (mail-yw1-f182.google.com [209.85.128.182]) by imf02.hostedemail.com (Postfix) with ESMTP id 6A90680004 for ; Fri, 9 Jun 2023 01:37:17 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="g/fK8hP5"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of hughd@google.com designates 209.85.128.182 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686274637; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zu44LOJFUZSfjUzXfaM7AXaMVzmcNQ9SKaG/n0S1Di4=; b=pNToysqZkdBUUfyvFlHLKWYkJyPvkrGeQCFBoY5R/EO/WbFHd/TQXY78440Sx6YvVELTqh ngQYWWh0xi7B/WFx2xV92EcCtVj87flsUepj5mw2efpqXhCbUSMiFeBPmZ07yrWZL/e0us UMbb3ZC5UbwX3jNQb6+0VrG6oJdb3Ms= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="g/fK8hP5"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of hughd@google.com designates 209.85.128.182 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686274637; a=rsa-sha256; cv=none; b=FKmRSy9JBLRxFYPGenNnQoLrXQ/Wbi5VrnePjvZkr5PiDAv29xlMAPlpvUdW2ZFpNzq3qr lch3sakKFVbXtAstLLwJkvAx85ckr+pSQHcQLqS5A4EZPxyS5N2eKmY1EfPnzNPlU3/9cm xC8Hkr6rzFcvwHACsEey/1lR7U/E8m4= Received: by mail-yw1-f182.google.com with SMTP id 00721157ae682-568ba7abc11so11571427b3.3 for ; Thu, 08 Jun 2023 18:37:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686274636; x=1688866636; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=zu44LOJFUZSfjUzXfaM7AXaMVzmcNQ9SKaG/n0S1Di4=; b=g/fK8hP5bFpFVjO76tLNArCNe2GlcEYd1TR7ZwQdCtSODKo9bn2ob62ekzm4ZsThhU yjQK+8xMaXjxfJ1yZWY7NOanC9yVcDVOaMe2iZYwQzT6eJ8h5uTrJC9oVwwCDxswbBHp l/xid7bUgRR1+bdWUBBbvmFDZ/iNYOS5AHsDau932Tn9EMZ4WD6ylqBkwxof8Mw7/egD Ww4k6FcpaXFX8Vqb/E2HUbm90tkOaEYqBxZ6KyMuDMKUc0iBxBMpA3o8pXAN+XBCXRzs f8EIBhd27TS5I+YyqHsJshT+x1siX1ieudcplgI00w6T7xrRyEgdPl7QxJSf2yBgKL3/ hKjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686274636; x=1688866636; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zu44LOJFUZSfjUzXfaM7AXaMVzmcNQ9SKaG/n0S1Di4=; b=LjTcuFtb+Hz9ytOjYHZduskHmnF9MoWjquhrDzrY0CwfYIOqxszQiQXoJMLXsquDil HahypTvFObkMw/cypZ6EYxXQL48qhu86ggd2fg2bDo2aTdJDCB/BBuAU+xEHIdKvYGTH in/3dnEABOKMrJNGxKNSQfIoGOCuh2rPpaQtkv9vXwtHkeyeggfcYnliTrl9U7PmeNEi oVQXPNKSUm2Mbc4OXbZ2ScpDXmDaW21UB/UYDfYGh1uirxQ5UcFFqr8wJ3wiTkHf8cRS AHbXBNxjlljPYN/5woR9+sTVV/CJBpwmZ9t4izOl7VidOU6YY6YP2dpCuurVTBXuN6CO Iyfw== X-Gm-Message-State: AC+VfDy1jCJ28nWedpNZHubiyyECYlKxLDpU5HYZowmBRPlpJ1Tc6oii T8g6W/KorVCubLwc/BS3FPZByQ== X-Google-Smtp-Source: ACHHUZ5BEYGpIOz49tbHOiOve9aTVph95zxtniJYmBvQce3mSMb/qY9GD2nlVpV9ClVLYloJKEZjzA== X-Received: by 2002:a81:46d7:0:b0:565:eedc:8dbe with SMTP id t206-20020a8146d7000000b00565eedc8dbemr1425297ywa.27.1686274636468; Thu, 08 Jun 2023 18:37:16 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id k124-20020a816f82000000b00565b26a9c9csm292968ywc.64.2023.06.08.18.37.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:37:15 -0700 (PDT) Date: Thu, 8 Jun 2023 18:37:12 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 23/32] mm/mglru: allow pte_offset_map_nolock() to fail In-Reply-To: Message-ID: <51ece73e-7398-2e4a-2384-56708c87844f@google.com> References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 6A90680004 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: d1ymmtk47srt16qftnaff9esb3wjh6c4 X-HE-Tag: 1686274637-422159 X-HE-Meta: U2FsdGVkX1/CZXJejtpF93+PmJHpAn9UKNCGrqSoi/dyNIPJTMo+qOuml/HRHqFW9ucXq06Aryw0fiZctX9Sh+91EBspSalOVjfndWQYVR3xlg/sj8FBxu+bvBc8vCaCCcuDUIy86xcj5tuGouPPw+G/vmFHJteEX6y4tZTLacaGyzC2H/GuSrvUdTzZi0Pyd/XbrFWZhJi27fvpUClMW1ihi5xgSNOIcbMbI/NxDYlgnljAdbcGdoCZDUnuE/yI8GzOIm1bfXjmz8oGa6HX6ZxtvtBzTnOSkWUhxlnVu2iGQMBpDpjetc79DfIec9iqk06y6OJNe+yGD+aTfRZBbtfAPvYK6lQMm/XnXoI7nwbRsFVwa+oJ5//DvsNuvoIav4jgkjofGZoIexNuz5TXucjBAG4oV+oTcWW9drQww1KDxEmCOw3+y9RFDcaMOEyS6uwFcA3hn+5Pc/ewhMEOYGg4VwXcTzy52Qcmp1qduVO5F7B2FNOJzKxB8sJcFGTUYx9LV59GrrnRuUZtoJ3SvjPGaweObejEdJGl+PIrCXyAOWbqEj2dgGOQFe52eFrczn9l/s9NMYmB9Y9kwn1Tx3vPC1PMnLrhncgZ2H4TFRPzQSQZZio+JVtTnPQoiA9BkaSxp+ieBjLjC1Kv0wGkZusvO3w+G4dft9A/4imf+wdEtRsu8rnobwalYhhN3j9mR28lfud+CvEnfhZkFIPZMdGNSPYn8fKy26P19Ilx4yaipGhx/iTdnjLoAuhyWR7q+oDIAHjk4mWh6KyCZL2isrpzwKVma0O2qcTNEj+4llqzJM7ST7cj/o5WlPyfrc2EeAIn8zGIdJjgQ4gXZU2W1/iYkNZJJLmo9f0jRmkRWT+KdpmF537XWUxSRNhtRzFr0UjU0kov6Nvik3w5vDWtfViZ3FBn3zpYk8QmDWyzpxHmR/4nLUygXBl0/JtuOcpj2wZiOs0KHJqITxNQF7v 8KUEM4w7 WudNeWReQtuiyKuvyK9v+XPRVW7cJ+D+QMxsOZa4859croYls/1qzR23k2rJPXiCBNDd1UaUnuiQ3k/1iJ1XE5XmE253qzGnz3wt9mT5fR0NVn4ZtjF6VK//DSfiZvHLRfNyMVq301P3xtdrd5F/t1HIHO6gAVjQcrvMtLxoVaVM7UMqRbyMJnYOkV6ne0fmcbK7E2Kc0Sfoj3oyRo127hXtaglme+flYlwtsBHyqgAWDL9Yid3hM8StHy7TSCrmfJcpAgS9dcxkgc32tdzxF0B8nG6OZDDw0ce6lTXYHtRDjEgnZmiAScJl+2kYC8U5/JDo/RV1vUtyUrJCmkwahuDvTJv069Dhld72edJrXRQR7jtz8cBTfAQmmXf+A00idSvMSMneGVUDxh3jCrHyFxRgDbOYakmcEZlYw1w+SXlbRQcG7jTCRI2f9og== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: MGLRU's walk_pte_range() use the safer pte_offset_map_nolock(), rather than pte_lockptr(), to get the ptl for its trylock. Just return false and move on to next extent if it fails, like when the trylock fails. Remove the VM_WARN_ON_ONCE(pmd_leaf) since that will happen, rarely. Signed-off-by: Hugh Dickins Acked-by: Yu Zhao --- mm/vmscan.c | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 6d0cd2840cf0..6a9bb6b30dc8 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3993,15 +3993,15 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, struct pglist_data *pgdat = lruvec_pgdat(walk->lruvec); int old_gen, new_gen = lru_gen_from_seq(walk->max_seq); - VM_WARN_ON_ONCE(pmd_leaf(*pmd)); - - ptl = pte_lockptr(args->mm, pmd); - if (!spin_trylock(ptl)) + pte = pte_offset_map_nolock(args->mm, pmd, start & PMD_MASK, &ptl); + if (!pte) return false; + if (!spin_trylock(ptl)) { + pte_unmap(pte); + return false; + } arch_enter_lazy_mmu_mode(); - - pte = pte_offset_map(pmd, start & PMD_MASK); restart: for (i = pte_index(start), addr = start; addr != end; i++, addr += PAGE_SIZE) { unsigned long pfn; @@ -4042,10 +4042,8 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, if (i < PTRS_PER_PTE && get_next_vma(PMD_MASK, PAGE_SIZE, args, &start, &end)) goto restart; - pte_unmap(pte); - arch_leave_lazy_mmu_mode(); - spin_unlock(ptl); + pte_unmap_unlock(pte, ptl); return suitable_to_scan(total, young); } From patchwork Fri Jun 9 01:38:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273089 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7FC81C7EE23 for ; Fri, 9 Jun 2023 01:38:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 23A398E0002; Thu, 8 Jun 2023 21:38:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1C3BF8E0001; Thu, 8 Jun 2023 21:38:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 03E288E0002; Thu, 8 Jun 2023 21:38:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E8B608E0001 for ; Thu, 8 Jun 2023 21:38:25 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id B8687402C8 for ; Fri, 9 Jun 2023 01:38:25 +0000 (UTC) X-FDA: 80881499370.02.D9C90B2 Received: from mail-ot1-f46.google.com (mail-ot1-f46.google.com [209.85.210.46]) by imf02.hostedemail.com (Postfix) with ESMTP id E35FB80004 for ; Fri, 9 Jun 2023 01:38:22 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=T0Kllypm; spf=pass (imf02.hostedemail.com: domain of hughd@google.com designates 209.85.210.46 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686274702; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AiDluOv2NKQXWITQOVsUWe3sj/TRHmtPFYOGjZ2ttyw=; b=nQO/M0vlJt9rmdYVnsSeO2oj3L+15Qjp//srfbDzHFKkOtnEnVynqrvVKHvBNRhfzZ2VT+ 6Fqtazt4pyAXSLOd9tdzFVWtt6K4e7HjAEqsKKnAng83oZfKCHJTjRFqr6yrrf95ldMICs jzH3JhsNTKBaqXVcvY8Ufja89QaAHGM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686274702; a=rsa-sha256; cv=none; b=PxkMrCJdZOU2xJry+Spc9xAvdn88H3SHEummk6JagQpVwe4CUZU7k3Udkq/f/K9+aJ6bfs jxJZFoOaYrbJUo1Vq98+W/qCIkMjPdhe3GdWKuDgk79ZCJzurxAiJ5pv/1P7uX7CiNAivj qJhAPCR2XxT5sjGB7/VmWxFXVvXZL1w= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=T0Kllypm; spf=pass (imf02.hostedemail.com: domain of hughd@google.com designates 209.85.210.46 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-ot1-f46.google.com with SMTP id 46e09a7af769-6b2993c9652so261145a34.1 for ; Thu, 08 Jun 2023 18:38:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686274702; x=1688866702; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=AiDluOv2NKQXWITQOVsUWe3sj/TRHmtPFYOGjZ2ttyw=; b=T0KllypmaI6dTB7DB6gV+sXkQkzh1ghuQsJWEK3jBazIFUwKZcsQ3r5kr988Onz36X Dex8mQRK1D2tkuguCbuqQKOnDZrak1tNbdPv9EQlJylAGS/JOPCnfCNAKd3nfZKgjEDX NyjKhyt6lzb+I77gA/PrGWpgqss/Ipf03liz3QsE5ZInJrgAzICnzUTYMrBYboMu8+Ko Te1BCCl46W+vNegr6tum9ZNWdColvrZcB8P0b/1QpqOrN9yiBM3NUya9DmMuMuxFqXqJ gRuJjfPoEiaWO0bbttWGNF01oOM4LC9iGoxIWBtycc73x4aBvX+fDbj/NGV2UJ/71lCf Nnag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686274702; x=1688866702; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=AiDluOv2NKQXWITQOVsUWe3sj/TRHmtPFYOGjZ2ttyw=; b=i0deFrEqpcK9NODVKs9hBt8ojwi0HeB1OvqtK2apfKX0v+/4rDTM1OgQzH6+FXRFqj K+6PS02M0sk/m7GAM1K6UBCh8KzYoSzq0SfSGw8TYZBcox2EfApN+lQTQFcXJ0D+huz1 eivMFYicitCtLLbUfR2s8fCh0uFwlqqr1JdcDM84HJJP1pbKMRPvxTeSzG0RSUF53YH9 HC2KXisEJ08jN8HogL/d8SJAwT0AJ3xxmkhyXDLH1qDbAHzVZUof0W9XX1emDi/vZHLQ 4K5r5rIwLfFGONYOkFgHYY7LhdnVeNzKrLU3SmyGKXB+bSTsjUCNOQsO08gBuHRgYMor vfUQ== X-Gm-Message-State: AC+VfDz4gtlevSRUE4l0t708xmZKUao3x9M3lI+Y443gi56RvMfg1IXK rf82QXspJ1E906AE3z/8WBzl/w== X-Google-Smtp-Source: ACHHUZ4lATUYWdzZUuFyIeEB4NQ1kndPyTKKCxuz2TahCj6Q7NKx5y9YMV8W26QVvIO6910YNrrjBg== X-Received: by 2002:a05:6830:cb:b0:6b1:570c:de5 with SMTP id x11-20020a05683000cb00b006b1570c0de5mr88477oto.17.1686274701910; Thu, 08 Jun 2023 18:38:21 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id y7-20020a056902052700b00b8f13ff2a8esm586262ybs.61.2023.06.08.18.38.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:38:21 -0700 (PDT) Date: Thu, 8 Jun 2023 18:38:17 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 24/32] mm/migrate_device: allow pte_offset_map_lock() to fail In-Reply-To: Message-ID: <1131be62-2e84-da2f-8f45-807b2cbeeec5@google.com> References: MIME-Version: 1.0 X-Rspamd-Queue-Id: E35FB80004 X-Rspam-User: X-Stat-Signature: jirz7yh1i3mub7137ysdz33aydmai3yn X-Rspamd-Server: rspam03 X-HE-Tag: 1686274702-146844 X-HE-Meta: U2FsdGVkX1+QLZ7FL2P5f2PExacbArNIxN4juwYPahfWYYV9u6jwbZ+j6vPXMrrT3VSBuhFl22pHFAWmabPw3wobQAZs5HoLcWfGi2yx6npfBFlgXklUfU0QePiuh+WcIWeNOcGQ/fhZRfZAbhOEyl82QilBOpXjt9AN99thsMhKWOTu7emuPMojpEKDLYH1Xcd+KnlsRtjBQdieuJ4rbKLiDXQ6yxoV00TNkjFCnA5QM4wDmclhBkJCbWy4Pbx4NS2QGzZG+h94FmsXOyd/6kaVu7ObEMt48Us8UCgCoIxzozl8ofctHMHVCxTDzkRUlqHShBG6HsiA0OXkZ8ywDwGb7MA6+wFxwrPU2KLVbaDPZ/F1menXCGOfNre4F7IEHyMiYVPjPc4PnVzbcAS5qJDalRUfSYUlDeVq0SgUbkvNxYlZJ9A2d1C/qwHR/GKyreA5fHcAcyPWsASaO5qu4ZgUdqD9VN1YuGtv7i3sh/Z1e0SNBT+6odUBMh4yWKpQxR5v+DKsmdEP/4OmQYgOqLcBUBGjxpXuNueXADjq2g8uEDtQcME6Lo9Z+keCniJbD1xBngeMyzsGSBB70mrH962P3ZnbatQnMv//sYFIhYJG9BpykiNh+ON1Xogv5ZguJfC0lcqHgpm9BXbR/wcz+zy8FiRfBlnLCi3RfgJ54zkA2HB3CdrKOtC68xLZfBD433thd8pZMDaA5V/xfKnyWUoMTF439VgdINmhHo5TlPsxSlf/+hj8pfwVUOOAkqkk0sUn3tC3CG9jutg5375e9G5TQgGwdtxdxgygxuvVdWHVIeEOA1D7STgBwk9doxLWVl7q7iEvXidUT6vv8jpnG7808BNN1kIXaWbJDw4aLUdnmHtX3xYnihOBKwtOnXgqjmpZksEr70NiaIMfYrVgs5HhV4UyO/CpC+oF65vUDpEO49moWZAenLN41rXJRj6iJ3YoNzy1RtZZuOdeFw0 pOSXV3bY eYOotFZQs5YHL8KI+BzKGMytQ+PWalYPX7fYOSpFuBqWVa3zP1OvX/6eattIoNJmxXdIfyprcd4zLI7Z9EZDW1onfDtX1c1dkB0oxNhr1Vbk/T2tpP0kMqP5axiBtTzBarMkm4SZMksANQvm6KTWV/rstdIrjdMMegxblGvTC9UdRG1jfjTW94DbyDGEvxxM8/wde5hgma6CRhRWGebXr9rtbrEezqQjVxAu+jOaXpm1VstXzl0Dvm+WQdt9uU7XscyVLSNcuV5u6BY1AAoE/DYno7rF78BYd7+HsdT5kA0+PV2qyhWuGm6erY63PDLB5sqfVrFs91kSkXS8jBc3HytzScedPPYfZueUwzPmjGJSF7p6KX1o+6AZgd56q3rnL0avo4+y2JjC/f5VbRKIcKk5xIbGGz0HMOgiLAyKu1rEZczmpcQQALBaQFqSMv+CiT5Zdf6D2Uekx4g8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: migrate_vma_collect_pmd(): remove the pmd_trans_unstable() handling after splitting huge zero pmd, and the pmd_none() handling after successfully splitting huge page: those are now managed inside pte_offset_map_lock(), and by "goto again" when it fails. But the skip after unsuccessful split_huge_page() must stay: it avoids an endless loop. The skip when pmd_bad()? Remove that: it will be treated as a hole rather than a skip once cleared by pte_offset_map_lock(), but with different timing that would be so anyway; and it's arguably best to leave the pmd_bad() handling centralized there. migrate_vma_insert_page(): remove comment on the old pte_offset_map() and old locking limitations; remove the pmd_trans_unstable() check and just proceed to pte_offset_map_lock(), aborting when it fails (page has been charged to memcg, but as in other cases, it's uncharged when freed). Signed-off-by: Hugh Dickins Reviewed-by: Alistair Popple --- mm/migrate_device.c | 31 ++++--------------------------- 1 file changed, 4 insertions(+), 27 deletions(-) diff --git a/mm/migrate_device.c b/mm/migrate_device.c index d30c9de60b0d..a14af6b12b04 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -83,9 +83,6 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, if (is_huge_zero_page(page)) { spin_unlock(ptl); split_huge_pmd(vma, pmdp, addr); - if (pmd_trans_unstable(pmdp)) - return migrate_vma_collect_skip(start, end, - walk); } else { int ret; @@ -100,16 +97,12 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, if (ret) return migrate_vma_collect_skip(start, end, walk); - if (pmd_none(*pmdp)) - return migrate_vma_collect_hole(start, end, -1, - walk); } } - if (unlikely(pmd_bad(*pmdp))) - return migrate_vma_collect_skip(start, end, walk); - ptep = pte_offset_map_lock(mm, pmdp, addr, &ptl); + if (!ptep) + goto again; arch_enter_lazy_mmu_mode(); for (; addr < end; addr += PAGE_SIZE, ptep++) { @@ -595,27 +588,10 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate, pmdp = pmd_alloc(mm, pudp, addr); if (!pmdp) goto abort; - if (pmd_trans_huge(*pmdp) || pmd_devmap(*pmdp)) goto abort; - - /* - * Use pte_alloc() instead of pte_alloc_map(). We can't run - * pte_offset_map() on pmds where a huge pmd might be created - * from a different thread. - * - * pte_alloc_map() is safe to use under mmap_write_lock(mm) or when - * parallel threads are excluded by other means. - * - * Here we only have mmap_read_lock(mm). - */ if (pte_alloc(mm, pmdp)) goto abort; - - /* See the comment in pte_alloc_one_map() */ - if (unlikely(pmd_trans_unstable(pmdp))) - goto abort; - if (unlikely(anon_vma_prepare(vma))) goto abort; if (mem_cgroup_charge(page_folio(page), vma->vm_mm, GFP_KERNEL)) @@ -650,7 +626,8 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate, } ptep = pte_offset_map_lock(mm, pmdp, addr, &ptl); - + if (!ptep) + goto abort; if (check_stable_address_space(mm)) goto unlock_abort; From patchwork Fri Jun 9 01:39:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273090 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CCE41C7EE23 for ; Fri, 9 Jun 2023 01:40:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6CD9F8E0002; Thu, 8 Jun 2023 21:40:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 67DE48E0001; Thu, 8 Jun 2023 21:40:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 51E5D8E0002; Thu, 8 Jun 2023 21:40:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 3E9438E0001 for ; Thu, 8 Jun 2023 21:40:07 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 1899A1602C7 for ; Fri, 9 Jun 2023 01:40:07 +0000 (UTC) X-FDA: 80881503654.15.0D4DAE3 Received: from mail-yb1-f172.google.com (mail-yb1-f172.google.com [209.85.219.172]) by imf17.hostedemail.com (Postfix) with ESMTP id 56D2040003 for ; Fri, 9 Jun 2023 01:40:05 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=xtJR0sJI; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of hughd@google.com designates 209.85.219.172 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686274805; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UwuVNK2fY7F7iaoH+tt2s4ap3NLwMgK8mAcGkUuV1QI=; b=7yTCsPdTQbS8Bmpsp68fLw6+6TdSZ6oKcuW6lUzXkTG0YIKeUeHcybBFWIvUoccN5aWvYd 9WaO1Fh1/BNce5NjftRmjsDoYGTXHZJLrcx8oLYvuuVFac6jzGQk19ubz0rXpr0F7ofgAS NhixDWSq0Nh0RvUMQ5qigFlRZ0w6o2M= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=xtJR0sJI; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of hughd@google.com designates 209.85.219.172 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686274805; a=rsa-sha256; cv=none; b=KQZ5Rem7hXqaJGzL3nZcNqFx49Cgk8QpmVW1VRvB0cwjtqsizKI2tCoS/Pa9vpXeO8L/u1 X4SdC8B2N3E5TVRF9qWeWT9lKzpagLWiwm/nh+51scTo+s4ycaV2oMSXzLelh12V0i+bF+ DFs2pNuzQ2DGuSM9r9yaAlojhxht2A8= Received: by mail-yb1-f172.google.com with SMTP id 3f1490d57ef6-ba841216e92so1373526276.1 for ; Thu, 08 Jun 2023 18:40:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686274804; x=1688866804; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=UwuVNK2fY7F7iaoH+tt2s4ap3NLwMgK8mAcGkUuV1QI=; b=xtJR0sJIJzLTXBky/wVlZo29dNTxzgA54BszS4z+bL4XgGa67uu8WNTm+JM7XI/l9k 6m4Lc80dzOLZH5qTELBlC5YxGQJLyz1bh8ZNW1tEJ8D6xjLzQ+u4+WAilxn55/jW0XRb Epze0ZHSodQnM7E7VM26/xnDCzhsJKvRWH4eXMeV0AMMehd09rxHboGfxvtXMtgKk1Z4 SnsamQypCiUjdvefoXG4abvyN6tDU3XIw1oNbEazjBhrMx+Kg+mHF/zkH+Ct85cC8hz5 iw1woW6T1GIXQ5NWzRO4+D6+WwHZuB7CFNtHE38x8bXk+v2L1K3lyzT2dxXGxy6c449H V+iA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686274804; x=1688866804; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UwuVNK2fY7F7iaoH+tt2s4ap3NLwMgK8mAcGkUuV1QI=; b=g5sZ8/tCXGookJ1aJCACudUfuqGZKpngieMqoNtP6X8//wa7p5+zegG8BwEa+ZMe1U p6cj+NPLJKiSv8+dDAdzHbgoxUluas9mWpn6pdqV+bdPpH173Wo6SlxYvDAkszNcxDqP VMRTf93rJjkTMLixgDhV2SOOJVekWQCymKQodfcTRryo6SDy99dpAhQQudv05xWiUMgh +qIoOEuVpzRvXTyK5/ETckFOZbt3UZZVZDKk7x2cKGU8HkDg+6ZjTaCAPjExfFyGWMIP at0moILnZLSnKsRIGiQZ6Qx7DlOFuXeYrzbqGGMjcaZOd9vDfo6Gmh2rzrb9ZuXDOlOB 85Og== X-Gm-Message-State: AC+VfDzJx88F1UOaaPkipfgmb/r2oe0cNNUrT26ynpUm8CfC0hF6B2nX qzFgiCIWd2cFhycVPIblNdrwzw== X-Google-Smtp-Source: ACHHUZ4q7dBjgPopIYtcorM5R7hyIrmP9CgiCOnb4btr582PPdFwrjvxsL7Fli0sAzIP7Ak/8SJsSA== X-Received: by 2002:a0d:f543:0:b0:561:ce93:b560 with SMTP id e64-20020a0df543000000b00561ce93b560mr1378798ywf.43.1686274804318; Thu, 08 Jun 2023 18:40:04 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id t202-20020a8183d3000000b005619cfb1b88sm306230ywf.52.2023.06.08.18.40.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:40:03 -0700 (PDT) Date: Thu, 8 Jun 2023 18:39:59 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 25/32] mm/gup: remove FOLL_SPLIT_PMD use of pmd_trans_unstable() In-Reply-To: Message-ID: <59fd15dd-4d39-5ec-2043-1d5117f7f85@google.com> References: MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: siitm36mecx1xeg36wg94kbubtqz7gqi X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 56D2040003 X-HE-Tag: 1686274805-84261 X-HE-Meta: U2FsdGVkX1+AFZEt7qXfIwp6EdkyC1uRxa/+Vnw8eo9r2IsqGkUUOB/4SP6IhT9Rn7QsIEX2Uw/Sdfbe7cW9R8N9ng76VMHHENQeDlMQfFZ2bufSXuzSXXw6atfsgaI2rCpU2Y9RE1eOf1NdjD9nJe8NO32WMGJ9+DAEapNxtr8yTioncEwPoPaIbcr7mRaBlfHUQCnRyhSY4yeFSfdDcyh0Tq2Hh6INq3e02Rh1jq/8FoK2fIJmyBtWpk4LAz51SVixY0CgsRhRb2I77hDGNg2N3r7Q+Yh3bFIHxgK1d3o3saspqZ41jRvoPFhV4u9GufhXGw2fxc7zjiZy87awZHjwftU7FU0ubaxGZjaOk/LwVWwdtkaTNlSDPPP20gzt6EF3SpSsLLNOHNME6ALJuvFHk2I0axhlg35bx5bc/ZGfCIFe0nnTKnNJXIPvRjvOmmj4RWnpvDERbwPMbIjtCGIHxW1qaYn4ydxjyN0s3Hl6rC1gkeM+oStJDraGGzlhH60aoa8abDB8X3geSSa2XkdbI9SFOQWCQS3FqIag5bU22pgtwSSZW/DcP4WL2/9SKrKj+pqerXbvnEaxflAv4lnsF6nNpCTFEcWLD62h/4YPow+K7RPEh5CYTlG3hFINAgLt25J8EYd0BtAhsFrwoTOaBqzTCGe1qTe2G31nERlR8aYZvLsDKlBqHpyO21vxmewSb6xQPjM2a81rhEpwpGq0k/ndp0OcCLFIZa0skKeTxlkP+MMJm8RmNMalq6kbVeiUhw8BK/tO+rn0zExP/Xezj6bUGCl/CN9Cm+9RHSOmPRyVfCPMvLcT+kGnqmmWFNPbsuWS1FHW2N+QRJwkX5cfycbKMK7dZtAJ/rqldYUpQYE4UCgFOX5YmsQzXdqFo6GnbD6plf1dsbpCUPP6AISGGDMzuhE58ZMlRHNSNO2DpD2GGP4QpNwDKWGhVJmbJDneiEXf+n9WkMiq4uA rFKhxKpd LRXBCPExYVwFlr4uz/OcfqviiFTN2CNWYLRGcfM7v/AGrck4X9GQ58PedXoxmfXUpecFvzz+pEmaCdkrwlqA0GQrLtASzUHiSXFMAxv1snQUloUyrnziVHL6fSyvYFTZQuF+HhAoovm1M61tnstrlsa+WZaxjPuHZfODeYYfNdfkU2Ycj+M1EzlFqSGwExrYs56D9Si+v2f3TfopFh77vLuUEV1OQMfqNUqtTIvhoJL/DMtqScoJLg1sE0vW/1O1on+kwkOLRwc50asACPQ0Dtyo+2oXpo3sSJ5qcUH2XzBqbihaB2+0J/QZ/eEBbnUxAn/XbCJgi8brxwjPNKXi6bzT2GomP1YYyW1VZSdNqeDyA7RQ6vxGxnGKc38IjEEMRlNnhiCrsLt7atOOiY0CenV54koK7cPJWMs3DDRZL0VZkJlbYTR2Os5HRyA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There is now no reason for follow_pmd_mask()'s FOLL_SPLIT_PMD block to distinguish huge_zero_page from a normal THP: follow_page_pte() handles any instability, and here it's a good idea to replace any pmd_none(*pmd) by a page table a.s.a.p, in the huge_zero_page case as for a normal THP; and this removes an unnecessary possibility of -EBUSY failure. (Hmm, couldn't the normal THP case have hit an unstably refaulted THP before? But there are only two, exceptional, users of FOLL_SPLIT_PMD.) Signed-off-by: Hugh Dickins Reviewed-by: Yang Shi --- mm/gup.c | 19 ++++--------------- 1 file changed, 4 insertions(+), 15 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index bb67193c5460..4ad50a59897f 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -681,21 +681,10 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, return follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); } if (flags & FOLL_SPLIT_PMD) { - int ret; - page = pmd_page(*pmd); - if (is_huge_zero_page(page)) { - spin_unlock(ptl); - ret = 0; - split_huge_pmd(vma, pmd, address); - if (pmd_trans_unstable(pmd)) - ret = -EBUSY; - } else { - spin_unlock(ptl); - split_huge_pmd(vma, pmd, address); - ret = pte_alloc(mm, pmd) ? -ENOMEM : 0; - } - - return ret ? ERR_PTR(ret) : + spin_unlock(ptl); + split_huge_pmd(vma, pmd, address); + /* If pmd was left empty, stuff a page table in there quickly */ + return pte_alloc(mm, pmd) ? ERR_PTR(-ENOMEM) : follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); } page = follow_trans_huge_pmd(vma, address, pmd, flags); From patchwork Fri Jun 9 01:41:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273092 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6118C7EE23 for ; Fri, 9 Jun 2023 01:41:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 853CD8E0002; Thu, 8 Jun 2023 21:41:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 804648E0001; Thu, 8 Jun 2023 21:41:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6F4658E0002; Thu, 8 Jun 2023 21:41:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 5F9848E0001 for ; Thu, 8 Jun 2023 21:41:39 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 31DED40287 for ; Fri, 9 Jun 2023 01:41:39 +0000 (UTC) X-FDA: 80881507518.23.A4E9E6D Received: from mail-yw1-f180.google.com (mail-yw1-f180.google.com [209.85.128.180]) by imf17.hostedemail.com (Postfix) with ESMTP id 5C3B24000F for ; Fri, 9 Jun 2023 01:41:37 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=ZUnTRtdJ; spf=pass (imf17.hostedemail.com: domain of hughd@google.com designates 209.85.128.180 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686274897; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9rqctUCF3S3JfLMZMEb80lefSOXfBunLGXTWgvpiAkU=; b=lwAjeQDGW+MJOoHcubP1mXoCtIwbEbpa3CPtDZuXNih6FW7R3B9niSxqjzDktpSCtKjyZ6 rnXkjzu8gXDPWU6Pw4gJCT1fTzktJN1yQrc7l5/HS6uqtP7mKKyxxB/7D5QgAzG/fXJfS+ IgZw+9g+m04WIUbPvAa+js8bfAARnR4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686274897; a=rsa-sha256; cv=none; b=n8/KMTnpSPq/ro+//cFrGMF6aRUUZ/eD2622lizXQ3DJmuWMtrQluuFnTfKvwh41022ODl E5f+qJxFuKWDDWA2kjVucEa1FvMRp9+sDArXhZ1rYngzHiMfOYv9sWqatG9p0ZLTMWJzfK dxq5gIPO6PuPDaVoHFk/xZwz+dR5eds= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=ZUnTRtdJ; spf=pass (imf17.hostedemail.com: domain of hughd@google.com designates 209.85.128.180 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f180.google.com with SMTP id 00721157ae682-565a6837a0bso11358797b3.3 for ; Thu, 08 Jun 2023 18:41:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686274896; x=1688866896; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=9rqctUCF3S3JfLMZMEb80lefSOXfBunLGXTWgvpiAkU=; b=ZUnTRtdJaKuP8HpCFk7QtfKJXS4h7TMBZvOJMvdH3Kb9GiIDkKt+HetF/u5hNY0iBZ C1Kvof77sPcx1H7YMQXI4g7GvGYj+dOG69GUGVHoheyoLtvIX7JK20pKUCPc88Hz5F7K 1vDSIi4a6aoe2Zceom9Svwsk2YImxUMqdpYy3MpT8APM++H9OeEHUgaErR5PTJoWvDnG Ooj+D4UNVVjEzEkwYMDebnqNIzPrU3Z0+Y3Vz1q/c+oe13fguEZcDxrw/xPw7f0EQgh9 2MxgqZ31UDTio0TPV7BeKiTntn6xKzAMDwB1lip/A72zUIdR7jZaKNlHZ1NNT8xv0XOp wchQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686274896; x=1688866896; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9rqctUCF3S3JfLMZMEb80lefSOXfBunLGXTWgvpiAkU=; b=TNWz94uoscg6mRDkRF50ECUNEtVNPsDfJRJ7BpvRArwQqcofbhz4nvFoNS0KYQCAX7 503h5fgJWHEPmklcOhNyO/TiQCtYeoSImcTVlOkVI74t724HD7yDGxs0BEf3HveF08Rw jyWrukc5U5HaKA17YT/Sl6ne7nHsQYPX9DVDmO2Ev7AolI9MrPevA2MZCZHWZrR1iDRm DH2+0qCib5xY51azx5RI/iiMxh6jUu+X0FMX1kFPcdUJNNgDhKGPzSyZ3zoHd2Etn4Xx mts5COwjlYarNZfLmxu38SDuPB7D4LtgAvPoA8AnOW2f2fFLo2THVBeWMOhtaP7Y/eyq Daow== X-Gm-Message-State: AC+VfDyOSL8C7huf/Jt7euZ65QGZV9l5voPDkiGXcFBwIQmMr8C6JvzA dcMHi8zqUasFwHOFkejL1yacGQ== X-Google-Smtp-Source: ACHHUZ4dqn2pyv+NnV1us+V+eC0MmFm8NCi0bji00YqYjgvIjBIANrUXMpCs8h/qJKTmj/xoG04trg== X-Received: by 2002:a81:6d04:0:b0:565:d3f9:209e with SMTP id i4-20020a816d04000000b00565d3f9209emr1374109ywc.34.1686274896335; Thu, 08 Jun 2023 18:41:36 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id r205-20020a0de8d6000000b00559d9989490sm304902ywe.41.2023.06.08.18.41.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:41:34 -0700 (PDT) Date: Thu, 8 Jun 2023 18:41:31 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 26/32] mm/huge_memory: split huge pmd under one pte_offset_map() In-Reply-To: Message-ID: <90cbed7f-90d9-b779-4a46-d2485baf9595@google.com> References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 5C3B24000F X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 7prksha9ia1w8bcodih7c3n655jgdkyx X-HE-Tag: 1686274897-945784 X-HE-Meta: U2FsdGVkX1/rNnU7MrzsMqTdxJjQ5Varxkeq4B8YXeLduc99smWwk8YnryCVboLDk+hyEf0uqDnEsKWAxxxWRktaR/6/89FDMQDaNvWU5k9xvtfKHwkeBGrfKRR2oT++BEZRPyERPIQ03jQor5MO35svRkIEOS9IHRwjVo0jq5SFtLWvvewmxdVkwcCiG3/freMTh/wRSFGg1SM0EGjrKVY4E4VewmXLsvDEmr2EzLFXXBA2pKHwhSwfPKugE4arL7OGnFt32TFOQRddEpQntomtQi57I36UaMqzAAP8x2qKbW0fYleR9BrMOQRJo2smh5JPC3jGt7PGeh2+t+N7Fbn8JNnhuACnsr54dOmkmcFgFkdzlRUN1YDK0HzXCOkw0i0xaj5qE6ZCiiYmBCQxVaBU7ZilmGRNXleidZesh0LY0AVnLUYIXkSPe76/SSr2mSkxCDXXf+Ks6Cb2ZUbgZoD8k1qYPpvpHHvCMQAm1ZueqdnLkA310st/lMLgVPgzspNb2x0WpMK7KvZTpjtX2se8UD3XWVHtE9cQRj+AZ+2/RFXeVcTEK2AVv7pwgn1Vbd7NV6E+GF7MOkLn9wxPrb/XAw91Em4aMcwogUN0IUPsqEj39pIcN+nQknZ35z3D139igO7SwDgx3s2R8Dm+zE+iINnqSR8GAAgLInJx2A3Qi0w0h8b47d2S51FtMagTYgvnld3mCpzOZuuQYktjN1CQPXYq+BMfTJJlDg/rBHX+V9Zm8S6/QCVN9wleA4Pr3xKUHP3iIaYzRs5JSqgNNWda3yJ1JN062kksmy4VcKdDF/hqTeKaoCDVMg1dgWgFJRDGh1nlkLLLQMDSdEgEOj/iP80PtzlHudA/9LbXi1+oB9dqv9YXNmlVVHAfGi8WiloUnvipXH10mcEzgAXq6hSuMhyYp9UFxZK+mgRTGh+4fAdtNmLDdn5nL+YuRowLTpeRsMpqmkSuxD1kJjT XHxM8Mne lLuW63DQ8yrMRcTrzRPTZ1BQPWzulF0J/6XgetAp65/pMcIrU+B2eDndhQjN+vgoiCvPwawK637+AJ0bbmRVnnsyFg6IgtYOa7x5Ii12BEfsidRTFtfA4k6MVp39VqkKD6vGNIc2xPEBF2c/2dQX0GNLV3XYyUA2wO4g3kwk+iTzN0uurTcQhK6Zi8LaYqHhiw2qV2S1b9xbAUsEyWSw2hnrTaZ/uzyrRlOECBT8WjelgoeeAoryL3GtjyYv4PqzxjUwemF87fDgX0rN0xXTqIGZJJ709cwUc8TPWwWo4jQwpU5xVWO9xzcWQHQzTDUIb+IHm9WPUSwIN3ci/vD/ALhmmlihTeZ9KhCWIDGIc39knn9Ztn2Nsyg2Zk+KlLZXaEQrwCkEyb4W67Cg5bMF69jiZfs3TT69RcWms/6Btsk2w8fvIYePgqRE47qMWrzFXyoBNyGaUoqU6jMU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: __split_huge_zero_page_pmd() use a single pte_offset_map() to sweep the extent: it's already under pmd_lock(), so this is no worse for latency; and since it's supposed to have full control of the just-withdrawn page table, here choose to VM_BUG_ON if it were to fail. And please don't increment haddr by PAGE_SIZE, that should remain huge aligned: declare a separate addr (not a bugfix, but it was deceptive). __split_huge_pmd_locked() likewise (but it had declared a separate addr); and change its BUG_ON(!pte_none) to VM_BUG_ON, for consistency with zero (those deposited page tables are sometimes victims of random corruption). Signed-off-by: Hugh Dickins Reviewed-by: Yang Shi --- mm/huge_memory.c | 28 ++++++++++++++++++---------- 1 file changed, 18 insertions(+), 10 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index d4bd5fa7c823..839c13fa0bbe 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2037,6 +2037,8 @@ static void __split_huge_zero_page_pmd(struct vm_area_struct *vma, struct mm_struct *mm = vma->vm_mm; pgtable_t pgtable; pmd_t _pmd, old_pmd; + unsigned long addr; + pte_t *pte; int i; /* @@ -2052,17 +2054,20 @@ static void __split_huge_zero_page_pmd(struct vm_area_struct *vma, pgtable = pgtable_trans_huge_withdraw(mm, pmd); pmd_populate(mm, &_pmd, pgtable); - for (i = 0; i < HPAGE_PMD_NR; i++, haddr += PAGE_SIZE) { - pte_t *pte, entry; - entry = pfn_pte(my_zero_pfn(haddr), vma->vm_page_prot); + pte = pte_offset_map(&_pmd, haddr); + VM_BUG_ON(!pte); + for (i = 0, addr = haddr; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE) { + pte_t entry; + + entry = pfn_pte(my_zero_pfn(addr), vma->vm_page_prot); entry = pte_mkspecial(entry); if (pmd_uffd_wp(old_pmd)) entry = pte_mkuffd_wp(entry); - pte = pte_offset_map(&_pmd, haddr); VM_BUG_ON(!pte_none(*pte)); - set_pte_at(mm, haddr, pte, entry); - pte_unmap(pte); + set_pte_at(mm, addr, pte, entry); + pte++; } + pte_unmap(pte - 1); smp_wmb(); /* make pte visible before pmd */ pmd_populate(mm, pmd, pgtable); } @@ -2077,6 +2082,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, bool young, write, soft_dirty, pmd_migration = false, uffd_wp = false; bool anon_exclusive = false, dirty = false; unsigned long addr; + pte_t *pte; int i; VM_BUG_ON(haddr & ~HPAGE_PMD_MASK); @@ -2205,8 +2211,10 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, pgtable = pgtable_trans_huge_withdraw(mm, pmd); pmd_populate(mm, &_pmd, pgtable); + pte = pte_offset_map(&_pmd, haddr); + VM_BUG_ON(!pte); for (i = 0, addr = haddr; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE) { - pte_t entry, *pte; + pte_t entry; /* * Note that NUMA hinting access restrictions are not * transferred to avoid any possibility of altering @@ -2249,11 +2257,11 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, entry = pte_mkuffd_wp(entry); page_add_anon_rmap(page + i, vma, addr, false); } - pte = pte_offset_map(&_pmd, addr); - BUG_ON(!pte_none(*pte)); + VM_BUG_ON(!pte_none(*pte)); set_pte_at(mm, addr, pte, entry); - pte_unmap(pte); + pte++; } + pte_unmap(pte - 1); if (!pmd_migration) page_remove_rmap(page, vma, true); From patchwork Fri Jun 9 01:42:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273093 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92025C7EE29 for ; Fri, 9 Jun 2023 01:42:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 34A608E0002; Thu, 8 Jun 2023 21:42:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2FA7B8E0001; Thu, 8 Jun 2023 21:42:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1C2BC8E0002; Thu, 8 Jun 2023 21:42:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 09F4B8E0001 for ; Thu, 8 Jun 2023 21:42:48 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id D3AE3A03AB for ; Fri, 9 Jun 2023 01:42:47 +0000 (UTC) X-FDA: 80881510374.16.726B18B Received: from mail-yb1-f169.google.com (mail-yb1-f169.google.com [209.85.219.169]) by imf21.hostedemail.com (Postfix) with ESMTP id 02F351C0004 for ; Fri, 9 Jun 2023 01:42:45 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=kXMH2hIQ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of hughd@google.com designates 209.85.219.169 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686274966; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=p1k5/eahjbWsfa/uahTffpcykz1TgCrP6tjOVjA4Rhk=; b=YIR0E3sy33xo7QRg2oH2mgAEXOVHWQxVpUOFY8MpboPGgG83PbKwSkKwdsWPqtFTOcNEEw 4+1XGOmLu05WpE2caAigP/1BfWfPuEXazJP8np3Xyqbmb2OfeHLlRjLhI3THUmF1VTEBhb usuU99AP+7rGuD8Aj9YGlTZufU0FD3o= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=kXMH2hIQ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of hughd@google.com designates 209.85.219.169 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686274966; a=rsa-sha256; cv=none; b=6Ljy9FVkyX5NgNkvaf5fnLXJsMZsKq2BWcuNMKk16JIMlE7wvkwHnLIfZFQTUDi1RDDvvX B7RASX6S4F6Mi8u1/ThYMWcymOXNERt0QDvm6jJFmgLEA8k7YCEOb1UQNMV+CkkAaNuLk5 bn6EOEn1vHWmGW9dKvIp8JMsGaZ9K0I= Received: by mail-yb1-f169.google.com with SMTP id 3f1490d57ef6-bad142be0ebso1284464276.3 for ; Thu, 08 Jun 2023 18:42:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686274965; x=1688866965; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=p1k5/eahjbWsfa/uahTffpcykz1TgCrP6tjOVjA4Rhk=; b=kXMH2hIQkmF2USapaQhiZcpKUKkNYVjPRafKmj8hC8uAM6lWt8Cg7WyWV3i5oVqEWA bi93PCMA7PjdAzeBItn2fVWVBY+/9qgI6tMfBLEeEpjUthSCUQOS9R5oolpPexMofMwP pSgpjbezGBNl2+s7mdLUqylcEDr0Rzwx/U2TjkwQG91WnUJpoHZuxPDmnmO/Yv7J8FdM cwm5pnD7LoAthDQwPzAWpV55QbduVNRTxBSVVH445+SgGv+mnH4NOD/8k4MMNgeNCGj5 /35hYwKb/TBzCQeOI3M6UpGu01ZJaZLpaHwXWmcS91MANZ+pzrD8lrdrhALbZ0SuH8aW G/vA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686274965; x=1688866965; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=p1k5/eahjbWsfa/uahTffpcykz1TgCrP6tjOVjA4Rhk=; b=A9tLEoUIF64Zn98AQqHwXQUB2vEFaVj1/4UhVka8yCz/tml8og0uUP+W3gpHaPYGGl qUo/0kboqrmY00NpUFW7hcv9smd0OKl6r1dlbN9NTB31+L78S5ODQUSV7OaBeyDJlrQs gKgOxY/Z4+Z7YS/81PhiIRW2iTIprB14D1dWZwo0LqfZOfDrUWjsHoG/aAb5SafMHJFL AR5W+LwQthLRi4GfHJLyCaDs5m7Z77SvRwF0FgEEXazVi4NTvuV2IMuepqx6Nj3TWfqP e/4NxeM/9em2HVm9rr6ZRQ3OIBsx+VDhGviU8wgO+bYVjXBSqQFemrh88ibEVOqOFlMd Lzkw== X-Gm-Message-State: AC+VfDzwCrc2IhwlDsl7Pgl+CGDz0UnRaOlveVm/I1f/Wu6PrGiWUHnQ OwYcIRW3721WyrPWaThQKNTFSw== X-Google-Smtp-Source: ACHHUZ5fn3kW9FeGHNpW1xzpcB4LGHphf3k1hTSF+y3EK3Em0GkiW50KAIuHoFANUJ/kQPH4MzH72g== X-Received: by 2002:a25:ab53:0:b0:ba8:972d:e380 with SMTP id u77-20020a25ab53000000b00ba8972de380mr1240110ybi.22.1686274964991; Thu, 08 Jun 2023 18:42:44 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id e188-20020a251ec5000000b00baca49c80dcsm615573ybe.28.2023.06.08.18.42.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:42:43 -0700 (PDT) Date: Thu, 8 Jun 2023 18:42:40 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 27/32] mm/khugepaged: allow pte_offset_map[_lock]() to fail In-Reply-To: Message-ID: <6513e85-d798-34ec-3762-7c24ffb9329@google.com> References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 02F351C0004 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: rxktgis1d44aizb53epyjrwj5jbg6t9g X-HE-Tag: 1686274965-797311 X-HE-Meta: U2FsdGVkX18tsV+L2MQRiohB7OIjPRooPxVOyzuO6eObJ6kUJ+5a8uweh6+e+i/0LpnceJhXS9YxBH/aiKwHCyg78IttaTIdp0IeibKE5ivs7WqaSSA1HeBD6QsmAhI9YvdxXoCb8kBzo0c1M/7auuz0MXmeBl0z5XnV8GJorNh9c7UJo0e+RUqfXN54v9v2MDc+ShTCw7OgoaTGxEqfzHAOyKPYBK9Gm5gY+w6ytYUItfAXbC+qJR7t3ePvugvrdrjfnQAcjleB4hHmqcXHV9Jut0/X2pk/xNvxuIi4lfeWQYs62rnZg3eDg8cdzYQloueKM/2TDlvSkj0bmqqa/mGcU2Ume6urLn1whuoxXpr4hhUUOJ0zw2IsiJD2S6Mqm9K5xHaA5RLd5YfKpanJINKAaiSim7+j6TwVBkc0zAoGm4aKtSd5iU0pher+jSd+u9PjMro+CZnTrvaXjz/A+QDwJmvv2/ODzqGydYBJoVsp6+etIn8nORIz5XlboQCzUo8aa1SEmKDIrYzJxUhc3TFiku+imF/IkrH/lQbR+Whd/yg42YHKEonld5zJjJQvFB97iU3kilnUnElHkWKpIbCAo98ODgSXEQZrX0hiAdW27XBX+pwxG5ICbG5aFwixfdiNg8AKC7bDhVP/tMqXC5eiMeYv5KFg4CJy5vFTO4gh9z4oPL8h+tq0gSeddqPz6Jf29lbYEZrrMjdgCIt0Mr83RYvblI8WqR0FaOKYj679YMMIx0JFmfXpamslG+bwMzsMrKhD73EMs+KLQhag6afnhbZEP+Vuoon2kAZ1mhpnEy+Yy00yxbl6xcf0Le1H7V1IIz5/BBfhSQVUMl4L+gJY3Q1BJrTFZXAhAdKQpUD6YS6n3tkSULqWwx4Ct8SA7kgILm1qrU5l3QpX9Pu16Me5C31gixgE+DlUKNuGmvLoKwXTxh6BYR2LBN+ZVxRJxacX4SLe/H6EpffRHqH itvQ1T6L iIydaXhpf5HNu3Lum/+bRj/kP5o8FGQkeyZQ01NMJbR+oyToWOMhprp51bJr/bDIY37ctCLMXccmdaSKheBbsT2uXNRhomeI3TmyWEOKt6IvjPp0WO1JYSOlp0eS+G53ogSkIQHpITcGKhtwgNZSdZzo4i1Gw5Eii+1RSzfA0DrQr7tAUwMWl6agnVo7xXOHEn4YPz5H+1fBZ9ESlJ5uXwaPKH1NNVh8vKw+K3FCFuzKOp9jrBRbT/X2c72TxDvUXvPmlQwgjw/9LwG64UX07eI7lXEoV9cquJ08yPvziuL9UuzqLlGA9HDR7MYbX9FrvNACplyo5BTA2F5T2+C3TKgcKzjaWDlDhvDu3vqG8/N78Cuqq7jXWPEAtxd+10u/3q9yEi1ZksG9oMzGdQNO8Bb0PTfF7gAhgVVy3McxIdemyg9BejnwflcAt3L/azrlWeJWRd00Ly/8mMS4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: __collapse_huge_page_swapin(): don't drop the map after every pte, it only has to be dropped by do_swap_page(); give up if pte_offset_map() fails; trace_mm_collapse_huge_page_swapin() at the end, with result; fix comment on returned result; fix vmf.pgoff, though it's not used. collapse_huge_page(): use pte_offset_map_lock() on the _pmd returned from clearing; allow failure, but it should be impossible there. hpage_collapse_scan_pmd() and collapse_pte_mapped_thp() allow for pte_offset_map_lock() failure. Signed-off-by: Hugh Dickins Reviewed-by: Yang Shi --- mm/khugepaged.c | 72 +++++++++++++++++++++++++++++++++---------------- 1 file changed, 49 insertions(+), 23 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 732f9ac393fc..49cfa7cdfe93 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -993,9 +993,8 @@ static int check_pmd_still_valid(struct mm_struct *mm, * Only done if hpage_collapse_scan_pmd believes it is worthwhile. * * Called and returns without pte mapped or spinlocks held. - * Note that if false is returned, mmap_lock will be released. + * Returns result: if not SCAN_SUCCEED, mmap_lock has been released. */ - static int __collapse_huge_page_swapin(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long haddr, pmd_t *pmd, @@ -1004,23 +1003,35 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm, int swapped_in = 0; vm_fault_t ret = 0; unsigned long address, end = haddr + (HPAGE_PMD_NR * PAGE_SIZE); + int result; + pte_t *pte = NULL; for (address = haddr; address < end; address += PAGE_SIZE) { struct vm_fault vmf = { .vma = vma, .address = address, - .pgoff = linear_page_index(vma, haddr), + .pgoff = linear_page_index(vma, address), .flags = FAULT_FLAG_ALLOW_RETRY, .pmd = pmd, }; - vmf.pte = pte_offset_map(pmd, address); - vmf.orig_pte = *vmf.pte; - if (!is_swap_pte(vmf.orig_pte)) { - pte_unmap(vmf.pte); - continue; + if (!pte++) { + pte = pte_offset_map(pmd, address); + if (!pte) { + mmap_read_unlock(mm); + result = SCAN_PMD_NULL; + goto out; + } } + + vmf.orig_pte = *pte; + if (!is_swap_pte(vmf.orig_pte)) + continue; + + vmf.pte = pte; ret = do_swap_page(&vmf); + /* Which unmaps pte (after perhaps re-checking the entry) */ + pte = NULL; /* * do_swap_page returns VM_FAULT_RETRY with released mmap_lock. @@ -1029,24 +1040,29 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm, * resulting in later failure. */ if (ret & VM_FAULT_RETRY) { - trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, 0); /* Likely, but not guaranteed, that page lock failed */ - return SCAN_PAGE_LOCK; + result = SCAN_PAGE_LOCK; + goto out; } if (ret & VM_FAULT_ERROR) { mmap_read_unlock(mm); - trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, 0); - return SCAN_FAIL; + result = SCAN_FAIL; + goto out; } swapped_in++; } + if (pte) + pte_unmap(pte); + /* Drain LRU add pagevec to remove extra pin on the swapped in pages */ if (swapped_in) lru_add_drain(); - trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, 1); - return SCAN_SUCCEED; + result = SCAN_SUCCEED; +out: + trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, result); + return result; } static int alloc_charge_hpage(struct page **hpage, struct mm_struct *mm, @@ -1146,9 +1162,6 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, address + HPAGE_PMD_SIZE); mmu_notifier_invalidate_range_start(&range); - pte = pte_offset_map(pmd, address); - pte_ptl = pte_lockptr(mm, pmd); - pmd_ptl = pmd_lock(mm, pmd); /* probably unnecessary */ /* * This removes any huge TLB entry from the CPU so we won't allow @@ -1163,13 +1176,18 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, mmu_notifier_invalidate_range_end(&range); tlb_remove_table_sync_one(); - spin_lock(pte_ptl); - result = __collapse_huge_page_isolate(vma, address, pte, cc, - &compound_pagelist); - spin_unlock(pte_ptl); + pte = pte_offset_map_lock(mm, &_pmd, address, &pte_ptl); + if (pte) { + result = __collapse_huge_page_isolate(vma, address, pte, cc, + &compound_pagelist); + spin_unlock(pte_ptl); + } else { + result = SCAN_PMD_NULL; + } if (unlikely(result != SCAN_SUCCEED)) { - pte_unmap(pte); + if (pte) + pte_unmap(pte); spin_lock(pmd_ptl); BUG_ON(!pmd_none(*pmd)); /* @@ -1253,6 +1271,11 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, memset(cc->node_load, 0, sizeof(cc->node_load)); nodes_clear(cc->alloc_nmask); pte = pte_offset_map_lock(mm, pmd, address, &ptl); + if (!pte) { + result = SCAN_PMD_NULL; + goto out; + } + for (_address = address, _pte = pte; _pte < pte + HPAGE_PMD_NR; _pte++, _address += PAGE_SIZE) { pte_t pteval = *_pte; @@ -1622,8 +1645,10 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, * lockless_pages_from_mm() and the hardware page walker can access page * tables while all the high-level locks are held in write mode. */ - start_pte = pte_offset_map_lock(mm, pmd, haddr, &ptl); result = SCAN_FAIL; + start_pte = pte_offset_map_lock(mm, pmd, haddr, &ptl); + if (!start_pte) + goto drop_immap; /* step 1: check all mapped PTEs are to the right huge page */ for (i = 0, addr = haddr, pte = start_pte; @@ -1697,6 +1722,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, abort: pte_unmap_unlock(start_pte, ptl); +drop_immap: i_mmap_unlock_write(vma->vm_file->f_mapping); goto drop_hpage; } From patchwork Fri Jun 9 01:43:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273094 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1C3DC7EE23 for ; Fri, 9 Jun 2023 01:43:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 62FB38E0002; Thu, 8 Jun 2023 21:43:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5E0318E0001; Thu, 8 Jun 2023 21:43:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4820A8E0002; Thu, 8 Jun 2023 21:43:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 35CCF8E0001 for ; Thu, 8 Jun 2023 21:43:46 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 06ABCA03AE for ; Fri, 9 Jun 2023 01:43:46 +0000 (UTC) X-FDA: 80881512852.20.BB437AF Received: from mail-yb1-f173.google.com (mail-yb1-f173.google.com [209.85.219.173]) by imf07.hostedemail.com (Postfix) with ESMTP id 384014001A for ; Fri, 9 Jun 2023 01:43:43 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="h7/lhgAB"; spf=pass (imf07.hostedemail.com: domain of hughd@google.com designates 209.85.219.173 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686275024; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NBl0ppCRA8l25HwCsBAYjZw+ZZ23COv4vEuhAo6MgQs=; b=FcGt7OZAbNXtVRIJUB1ML8yOaRkBZfBXzWnOrC9xLDkb2dctTqir610k+MW9D5mPc+eyiN KU4CZg2Hyv644OF+UpHE8vkUL/uInwHkONz2OKidH3jucJDbrT3QOVep8Xj4ZWxA4kZPUx Xr5LjGreSTn/OxCIOd7aoRFZ7Q9Oq0g= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686275024; a=rsa-sha256; cv=none; b=XKNSxbPJvGGSE71DCuzQ2rDfn20YXFJzDojrLrqlw6YcOs6Fs5w8JI4YKUWtiItU/Aq0fG xVbdGmB9vkFkJCESih7Dt7ClQKsaci5Jw2awpbwdbqcKh3D6jufh2+Jqg/OhcBNWkipxE1 d6OUiIlVaporKa7WdjVgE0hC+9Ith7Q= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="h7/lhgAB"; spf=pass (imf07.hostedemail.com: domain of hughd@google.com designates 209.85.219.173 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f173.google.com with SMTP id 3f1490d57ef6-bacfb7acdb7so1342516276.0 for ; Thu, 08 Jun 2023 18:43:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686275023; x=1688867023; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=NBl0ppCRA8l25HwCsBAYjZw+ZZ23COv4vEuhAo6MgQs=; b=h7/lhgABcsEuN80ZtY1CkQPEyiRM02tdU0Fj4Bqu3XEJ6+oMt1JttOuF1+QhBlyKlU nItRcb1lhVcg3eXONIukosC/ovw61FcNIrky9ZYjzzwajHv6HsTcvoIBjY3VhzMNyh1o MljGnp6iHoyALEuaRGEygREAs7NiPhq7Ts0bzgZaUDe8a2nZJcIQ2+Lj1wkKmKdUDbGy NH5AIRGaz2lGcsHiKp41MEO/swZqDgq35wqrQHtww6/nIP6oqDLpTQPv1VdUEAb8WQlv b+6qKzgTWVc0T+xle5LZMl0lKQUz8B9zZKcS4HbNa7M3nikEfj9E/klPVWnDJWzUyVm/ ISKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686275023; x=1688867023; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=NBl0ppCRA8l25HwCsBAYjZw+ZZ23COv4vEuhAo6MgQs=; b=aBI9UylUDNUS9PmI6MosqfhscjzYV3XZh2/yCb7QtMJ0tBmusagG3VIjegsqZwWo5N HEW2nYHHvTTxGzGNmnmmMnF8cPU8qSQ0Y4rHPJCQHvZElhcM+1UOoEJpUlXAubUsfRaJ OfE+ajnyDvGWc7ShS4Is4JJjXLQKiqkUSwWIOd4MkVVtORZyiQ7tZRO50endhdqi8x6p VtomuRp5TYGWYfzJp23CkUBj0XOmrOKmf+8gnWSi+0FpIoi6BaI9c4+/H/0RY3hbQ/Nr vCmkYNb0y9noP2Q2Ef5YWzKqRROiqpraAG1TAlBDgGVmwObppHJ55SYqs/sXDv7Ea3wo nDNA== X-Gm-Message-State: AC+VfDx7RVDJkzWk931Wk54ExanNzcXxa5+b3xZa1StWHBy7VowWXb5w tViNrjEGHh+k9uIQNxDrMgK/Ig== X-Google-Smtp-Source: ACHHUZ7uNPgy7ZBXRocyBo3vsfUWGTN3VMWhF4yrbhs6dEiMkK2v8+kmP/oHR3LauFiKONeXt7BBYg== X-Received: by 2002:a0d:f6c4:0:b0:55a:40d3:4d6f with SMTP id g187-20020a0df6c4000000b0055a40d34d6fmr1156326ywf.26.1686275023013; Thu, 08 Jun 2023 18:43:43 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id a17-20020a81bb51000000b00545a08184fdsm281040ywl.141.2023.06.08.18.43.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:43:41 -0700 (PDT) Date: Thu, 8 Jun 2023 18:43:38 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 28/32] mm/memory: allow pte_offset_map[_lock]() to fail In-Reply-To: Message-ID: References: MIME-Version: 1.0 X-Stat-Signature: 8c1ff3kjcfzzirf9apa5qipxz5s35tz1 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 384014001A X-Rspam-User: X-HE-Tag: 1686275023-664687 X-HE-Meta: U2FsdGVkX1/aHTGM3fyQkgibezir8cMmCiOxEMIGF2V4koWxVgD9Y5+FomZgyyjplyOaCjr9yWgRGTSsNj44J4fQuErwNjWB1zrnUAN/Y5aVaT3ZwMKeBdtZd+h6EsFQhc15BEW3+AnN7nevDoo1nCOVgxNQmnWWrGaQ+o2WuntYqTLq9fz4linU/oPzIzGUHIkLGSd0XnhvDJ7SQXrXjkJQPS83b1SVUN+OpcBQmNI+xEzQ/hHso+rpAng9aPvZqzDknUuCutO/43ahGmds+n6pGC76VCJ3788sasWl4CLZt/3fx3+CqK2klcanWRm19cRf8GktdN4oNOO2IpZa6smLHXCCoYXeo/9HwrOPZodCWc0AfznDEPZvWszBcrkA4L4Wm96Z9DhtOh0BujfYTLrouCwypz4ctwJeQeZK3vLTkQWM1VtoZamsfgO3YZX0wxtaQ+UM1TNwIbMzp7d+g8LbHk852K8Ag/Ho+tPpUXquQ38+HfbuXN1Lxxj+aHK7CGtxI1tWl4BZgB4Asnqw3d9KBQxtPgzHe0RjsSWbqkDy3Oph7pHaXmCV28TEjEfpSFxRinZk8nbXHVNMUV195gN7fSyjkXTx38+8B3HAa8nw8AYvugGFw56nYAf7ZyXBDelbY2xj2rtvuB5alOM24+60x9T3ewWlYRP+dRjUZbP2LHW8tKPXLQaXJLDZ+rajU9maaDbYsxAewPXQ5Cf/bNZdINButwoUwkGUjyHlH3pEapDXg3RWczqA3qMLqOoNImoXbymICNkcoj9ZfYVF3iavXXaTbywLDyv6XOGdHWGH5cHT4vh52qYpVpxprImQJFu/TmRPSjzUgtZYi7CkIvhM00tEcQGj507QrnbAPgWZmz72yT714CgpmAQswNB41WbMulvJcQBOQbocQkBI2bL36LtPHbqHwcpftVm57nHwm228oycBs7fkqvXQFHG6L/EBu1JgCaChN8BKnsW OibsuV46 AeY5nzJWwNwgp6RvCF5QcuwFoczvGtDa+H8EfQ//Gy7+DF7TCBMZ157WPjUcgNJxQow4oLX25WzVxK7gDAJAy81uWdBpR1QVn0D9GmcHHyrKpFKfjgZdfg+ntKaflSC6t2bTVCmoXJO5FdeL3dbB4J1T2WYP3TCiENWIWrjtQUF41prjnDhtmi/WMkYgqNPjFR8z+tgi3UPyKt88JTMyQVCbBSvqRveGGAfVyzob8yHMrzPXWopk5mAF/1AZRudw7fUEcQWSjlcNaemF2runZrdiXOJUG2iJCJd4VTvoNBx+5gXhLJScmWT9qRQ8s1Y5yHGuZ0bhT/oFZ2lIHSj4kXSeIK6EKYCtOkFPvhpVoEUQ7YF4ZsBjXO/COsF8E1WgXXHLjTbX8QCuR86JgFlSV3HUi6Y2nuOSPSXws/hmA6+bBybNF4zUeN4ARxQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: copy_pte_range(): use pte_offset_map_nolock(), and allow for it to fail; but with a comment on some further assumptions that are being made there. zap_pte_range() and zap_pmd_range(): adjust their interaction so that a pte_offset_map_lock() failure in zap_pte_range() leads to a retry in zap_pmd_range(); remove call to pmd_none_or_trans_huge_or_clear_bad(). Allow pte_offset_map_lock() to fail in many functions. Update comment on calling pte_alloc() in do_anonymous_page(). Remove redundant calls to pmd_trans_unstable(), pmd_devmap_trans_unstable(), pmd_none() and pmd_bad(); but leave pmd_none_or_clear_bad() calls in free_pmd_range() and copy_pmd_range(), those do simplify the next level down. Signed-off-by: Hugh Dickins --- mm/memory.c | 172 +++++++++++++++++++++++++--------------------------- 1 file changed, 82 insertions(+), 90 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 2eb54c0d5d3c..c7b920291a72 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1012,13 +1012,25 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, progress = 0; init_rss_vec(rss); + /* + * copy_pmd_range()'s prior pmd_none_or_clear_bad(src_pmd), and the + * error handling here, assume that exclusive mmap_lock on dst and src + * protects anon from unexpected THP transitions; with shmem and file + * protected by mmap_lock-less collapse skipping areas with anon_vma + * (whereas vma_needs_copy() skips areas without anon_vma). A rework + * can remove such assumptions later, but this is good enough for now. + */ dst_pte = pte_alloc_map_lock(dst_mm, dst_pmd, addr, &dst_ptl); if (!dst_pte) { ret = -ENOMEM; goto out; } - src_pte = pte_offset_map(src_pmd, addr); - src_ptl = pte_lockptr(src_mm, src_pmd); + src_pte = pte_offset_map_nolock(src_mm, src_pmd, addr, &src_ptl); + if (!src_pte) { + pte_unmap_unlock(dst_pte, dst_ptl); + /* ret == 0 */ + goto out; + } spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); orig_src_pte = src_pte; orig_dst_pte = dst_pte; @@ -1083,8 +1095,7 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, } while (dst_pte++, src_pte++, addr += PAGE_SIZE, addr != end); arch_leave_lazy_mmu_mode(); - spin_unlock(src_ptl); - pte_unmap(orig_src_pte); + pte_unmap_unlock(orig_src_pte, src_ptl); add_mm_rss_vec(dst_mm, rss); pte_unmap_unlock(orig_dst_pte, dst_ptl); cond_resched(); @@ -1388,10 +1399,11 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, swp_entry_t entry; tlb_change_page_size(tlb, PAGE_SIZE); -again: init_rss_vec(rss); - start_pte = pte_offset_map_lock(mm, pmd, addr, &ptl); - pte = start_pte; + start_pte = pte = pte_offset_map_lock(mm, pmd, addr, &ptl); + if (!pte) + return addr; + flush_tlb_batched_pending(mm); arch_enter_lazy_mmu_mode(); do { @@ -1507,17 +1519,10 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, * If we forced a TLB flush (either due to running out of * batch buffers or because we needed to flush dirty TLB * entries before releasing the ptl), free the batched - * memory too. Restart if we didn't do everything. + * memory too. Come back again if we didn't do everything. */ - if (force_flush) { - force_flush = 0; + if (force_flush) tlb_flush_mmu(tlb); - } - - if (addr != end) { - cond_resched(); - goto again; - } return addr; } @@ -1536,8 +1541,10 @@ static inline unsigned long zap_pmd_range(struct mmu_gather *tlb, if (is_swap_pmd(*pmd) || pmd_trans_huge(*pmd) || pmd_devmap(*pmd)) { if (next - addr != HPAGE_PMD_SIZE) __split_huge_pmd(vma, pmd, addr, false, NULL); - else if (zap_huge_pmd(tlb, vma, pmd, addr)) - goto next; + else if (zap_huge_pmd(tlb, vma, pmd, addr)) { + addr = next; + continue; + } /* fall through */ } else if (details && details->single_folio && folio_test_pmd_mappable(details->single_folio) && @@ -1550,20 +1557,14 @@ static inline unsigned long zap_pmd_range(struct mmu_gather *tlb, */ spin_unlock(ptl); } - - /* - * Here there can be other concurrent MADV_DONTNEED or - * trans huge page faults running, and if the pmd is - * none or trans huge it can change under us. This is - * because MADV_DONTNEED holds the mmap_lock in read - * mode. - */ - if (pmd_none_or_trans_huge_or_clear_bad(pmd)) - goto next; - next = zap_pte_range(tlb, vma, pmd, addr, next, details); -next: - cond_resched(); - } while (pmd++, addr = next, addr != end); + if (pmd_none(*pmd)) { + addr = next; + continue; + } + addr = zap_pte_range(tlb, vma, pmd, addr, next, details); + if (addr != next) + pmd--; + } while (pmd++, cond_resched(), addr != end); return addr; } @@ -1905,6 +1906,10 @@ static int insert_pages(struct vm_area_struct *vma, unsigned long addr, const int batch_size = min_t(int, pages_to_write_in_pmd, 8); start_pte = pte_offset_map_lock(mm, pmd, addr, &pte_lock); + if (!start_pte) { + ret = -EFAULT; + goto out; + } for (pte = start_pte; pte_idx < batch_size; ++pte, ++pte_idx) { int err = insert_page_in_batch_locked(vma, pte, addr, pages[curr_page_idx], prot); @@ -2572,10 +2577,10 @@ static int apply_to_pte_range(struct mm_struct *mm, pmd_t *pmd, mapped_pte = pte = (mm == &init_mm) ? pte_offset_kernel(pmd, addr) : pte_offset_map_lock(mm, pmd, addr, &ptl); + if (!pte) + return -EINVAL; } - BUG_ON(pmd_huge(*pmd)); - arch_enter_lazy_mmu_mode(); if (fn) { @@ -2804,7 +2809,6 @@ static inline int __wp_page_copy_user(struct page *dst, struct page *src, int ret; void *kaddr; void __user *uaddr; - bool locked = false; struct vm_area_struct *vma = vmf->vma; struct mm_struct *mm = vma->vm_mm; unsigned long addr = vmf->address; @@ -2830,12 +2834,12 @@ static inline int __wp_page_copy_user(struct page *dst, struct page *src, * On architectures with software "accessed" bits, we would * take a double page fault, so mark it accessed here. */ + vmf->pte = NULL; if (!arch_has_hw_pte_young() && !pte_young(vmf->orig_pte)) { pte_t entry; vmf->pte = pte_offset_map_lock(mm, vmf->pmd, addr, &vmf->ptl); - locked = true; - if (!likely(pte_same(*vmf->pte, vmf->orig_pte))) { + if (unlikely(!vmf->pte || !pte_same(*vmf->pte, vmf->orig_pte))) { /* * Other thread has already handled the fault * and update local tlb only @@ -2857,13 +2861,12 @@ static inline int __wp_page_copy_user(struct page *dst, struct page *src, * zeroes. */ if (__copy_from_user_inatomic(kaddr, uaddr, PAGE_SIZE)) { - if (locked) + if (vmf->pte) goto warn; /* Re-validate under PTL if the page is still mapped */ vmf->pte = pte_offset_map_lock(mm, vmf->pmd, addr, &vmf->ptl); - locked = true; - if (!likely(pte_same(*vmf->pte, vmf->orig_pte))) { + if (unlikely(!vmf->pte || !pte_same(*vmf->pte, vmf->orig_pte))) { /* The PTE changed under us, update local tlb */ update_mmu_tlb(vma, addr, vmf->pte); ret = -EAGAIN; @@ -2888,7 +2891,7 @@ static inline int __wp_page_copy_user(struct page *dst, struct page *src, ret = 0; pte_unlock: - if (locked) + if (vmf->pte) pte_unmap_unlock(vmf->pte, vmf->ptl); kunmap_atomic(kaddr); flush_dcache_page(dst); @@ -3110,7 +3113,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) * Re-check the pte - we dropped the lock */ vmf->pte = pte_offset_map_lock(mm, vmf->pmd, vmf->address, &vmf->ptl); - if (likely(pte_same(*vmf->pte, vmf->orig_pte))) { + if (likely(vmf->pte && pte_same(*vmf->pte, vmf->orig_pte))) { if (old_folio) { if (!folio_test_anon(old_folio)) { dec_mm_counter(mm, mm_counter_file(&old_folio->page)); @@ -3178,19 +3181,20 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) /* Free the old page.. */ new_folio = old_folio; page_copied = 1; - } else { + pte_unmap_unlock(vmf->pte, vmf->ptl); + } else if (vmf->pte) { update_mmu_tlb(vma, vmf->address, vmf->pte); + pte_unmap_unlock(vmf->pte, vmf->ptl); } - if (new_folio) - folio_put(new_folio); - - pte_unmap_unlock(vmf->pte, vmf->ptl); /* * No need to double call mmu_notifier->invalidate_range() callback as * the above ptep_clear_flush_notify() did already call it. */ mmu_notifier_invalidate_range_only_end(&range); + + if (new_folio) + folio_put(new_folio); if (old_folio) { if (page_copied) free_swap_cache(&old_folio->page); @@ -3230,6 +3234,8 @@ vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf) WARN_ON_ONCE(!(vmf->vma->vm_flags & VM_SHARED)); vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); + if (!vmf->pte) + return VM_FAULT_NOPAGE; /* * We might have raced with another page fault while we released the * pte_offset_map_lock. @@ -3591,10 +3597,11 @@ static vm_fault_t remove_device_exclusive_entry(struct vm_fault *vmf) vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); - if (likely(pte_same(*vmf->pte, vmf->orig_pte))) + if (likely(vmf->pte && pte_same(*vmf->pte, vmf->orig_pte))) restore_exclusive_pte(vma, vmf->page, vmf->address, vmf->pte); - pte_unmap_unlock(vmf->pte, vmf->ptl); + if (vmf->pte) + pte_unmap_unlock(vmf->pte, vmf->ptl); folio_unlock(folio); folio_put(folio); @@ -3625,6 +3632,8 @@ static vm_fault_t pte_marker_clear(struct vm_fault *vmf) { vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); + if (!vmf->pte) + return 0; /* * Be careful so that we will only recover a special uffd-wp pte into a * none pte. Otherwise it means the pte could have changed, so retry. @@ -3728,11 +3737,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) vmf->page = pfn_swap_entry_to_page(entry); vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); - if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte))) { - spin_unlock(vmf->ptl); - goto out; - } - + if (unlikely(!vmf->pte || + !pte_same(*vmf->pte, vmf->orig_pte))) + goto unlock; /* * Get a page reference while we know the page can't be * freed. @@ -3807,7 +3814,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) */ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); - if (likely(pte_same(*vmf->pte, vmf->orig_pte))) + if (likely(vmf->pte && pte_same(*vmf->pte, vmf->orig_pte))) ret = VM_FAULT_OOM; goto unlock; } @@ -3877,7 +3884,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) */ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); - if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte))) + if (unlikely(!vmf->pte || !pte_same(*vmf->pte, vmf->orig_pte))) goto out_nomap; if (unlikely(!folio_test_uptodate(folio))) { @@ -4003,13 +4010,15 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) /* No need to invalidate - it was non-present before */ update_mmu_cache(vma, vmf->address, vmf->pte); unlock: - pte_unmap_unlock(vmf->pte, vmf->ptl); + if (vmf->pte) + pte_unmap_unlock(vmf->pte, vmf->ptl); out: if (si) put_swap_device(si); return ret; out_nomap: - pte_unmap_unlock(vmf->pte, vmf->ptl); + if (vmf->pte) + pte_unmap_unlock(vmf->pte, vmf->ptl); out_page: folio_unlock(folio); out_release: @@ -4041,22 +4050,12 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) return VM_FAULT_SIGBUS; /* - * Use pte_alloc() instead of pte_alloc_map(). We can't run - * pte_offset_map() on pmds where a huge pmd might be created - * from a different thread. - * - * pte_alloc_map() is safe to use under mmap_write_lock(mm) or when - * parallel threads are excluded by other means. - * - * Here we only have mmap_read_lock(mm). + * Use pte_alloc() instead of pte_alloc_map(), so that OOM can + * be distinguished from a transient failure of pte_offset_map(). */ if (pte_alloc(vma->vm_mm, vmf->pmd)) return VM_FAULT_OOM; - /* See comment in handle_pte_fault() */ - if (unlikely(pmd_trans_unstable(vmf->pmd))) - return 0; - /* Use the zero-page for reads */ if (!(vmf->flags & FAULT_FLAG_WRITE) && !mm_forbids_zeropage(vma->vm_mm)) { @@ -4064,6 +4063,8 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) vma->vm_page_prot)); vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); + if (!vmf->pte) + goto unlock; if (vmf_pte_changed(vmf)) { update_mmu_tlb(vma, vmf->address, vmf->pte); goto unlock; @@ -4104,6 +4105,8 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); + if (!vmf->pte) + goto release; if (vmf_pte_changed(vmf)) { update_mmu_tlb(vma, vmf->address, vmf->pte); goto release; @@ -4131,7 +4134,8 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) /* No need to invalidate - it was non-present before */ update_mmu_cache(vma, vmf->address, vmf->pte); unlock: - pte_unmap_unlock(vmf->pte, vmf->ptl); + if (vmf->pte) + pte_unmap_unlock(vmf->pte, vmf->ptl); return ret; release: folio_put(folio); @@ -4380,15 +4384,10 @@ vm_fault_t finish_fault(struct vm_fault *vmf) return VM_FAULT_OOM; } - /* - * See comment in handle_pte_fault() for how this scenario happens, we - * need to return NOPAGE so that we drop this page. - */ - if (pmd_devmap_trans_unstable(vmf->pmd)) - return VM_FAULT_NOPAGE; - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); + if (!vmf->pte) + return VM_FAULT_NOPAGE; /* Re-check under ptl */ if (likely(!vmf_pte_changed(vmf))) { @@ -4630,17 +4629,11 @@ static vm_fault_t do_fault(struct vm_fault *vmf) * The VMA was not fully populated on mmap() or missing VM_DONTEXPAND */ if (!vma->vm_ops->fault) { - /* - * If we find a migration pmd entry or a none pmd entry, which - * should never happen, return SIGBUS - */ - if (unlikely(!pmd_present(*vmf->pmd))) + vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, + vmf->address, &vmf->ptl); + if (unlikely(!vmf->pte)) ret = VM_FAULT_SIGBUS; else { - vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, - vmf->pmd, - vmf->address, - &vmf->ptl); /* * Make sure this is not a temporary clearing of pte * by holding ptl and checking again. A R/M/W update @@ -5429,10 +5422,9 @@ int follow_pte(struct mm_struct *mm, unsigned long address, pmd = pmd_offset(pud, address); VM_BUG_ON(pmd_trans_huge(*pmd)); - if (pmd_none(*pmd) || unlikely(pmd_bad(*pmd))) - goto out; - ptep = pte_offset_map_lock(mm, pmd, address, ptlp); + if (!ptep) + goto out; if (!pte_present(*ptep)) goto unlock; *ptepp = ptep; From patchwork Fri Jun 9 01:45:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273095 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57291C7EE29 for ; Fri, 9 Jun 2023 01:45:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DA9D28E0003; Thu, 8 Jun 2023 21:45:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D80868E0001; Thu, 8 Jun 2023 21:45:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C487D8E0003; Thu, 8 Jun 2023 21:45:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id B63858E0001 for ; Thu, 8 Jun 2023 21:45:12 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 8229F80275 for ; Fri, 9 Jun 2023 01:45:12 +0000 (UTC) X-FDA: 80881516464.06.ECC4E68 Received: from mail-yb1-f177.google.com (mail-yb1-f177.google.com [209.85.219.177]) by imf25.hostedemail.com (Postfix) with ESMTP id AD2D1A0009 for ; Fri, 9 Jun 2023 01:45:10 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=5nDVvDFD; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of hughd@google.com designates 209.85.219.177 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686275110; a=rsa-sha256; cv=none; b=zJf2ZJgLECPNlUh1Y+rLi3mTgr2yFXQ7phyb9YBN3FQD0Qi8eK3anNbgjT/99uF+FoYOE2 ANH9sh6bHd0qGibDQk4TPNs/wGejNojwBOEYlEupML98BqGM9BDsnRFPUoC7HkiLGSoTpn /yoghyvrE77yd4GTAE/jQ3wY8a2fVVU= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=5nDVvDFD; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of hughd@google.com designates 209.85.219.177 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686275110; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6V2+W0bna42qnZ43tvOf68yFS9Jt1uoq2b6WWLaZ+Rg=; b=PpnVUezmgB3CPphipXOK7cLik0cXbOWYScvcnLo6CS87HiFdBVxo5nwZBDRIyLk6fQdkdr zceoF3Plkmgw8zdof1Uo8cQ04OouNiMxsQ2tgIZLwCe4I35bZt/NGelv14RbVmrDmfgIkE lUeFre6LGdblzKB3EAAucziBRHAQHF8= Received: by mail-yb1-f177.google.com with SMTP id 3f1490d57ef6-bacf9edc87bso1268526276.1 for ; Thu, 08 Jun 2023 18:45:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686275110; x=1688867110; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=6V2+W0bna42qnZ43tvOf68yFS9Jt1uoq2b6WWLaZ+Rg=; b=5nDVvDFDj0TDSwA6VyREm/b7XZyfd9szkT+LB84tX5XiJW9/M17p2pjZw/Ek5jNCOs 40hFeMoNWqiJJMeNTMe2RktTogEBFqE0QPIaTtgF826N1Ako6QEnbFjFhp7dOZXF3sOJ aEBqtDp9A7OZ2dPSncNi3eN5NLAJGJkMJEjCptn2bKCi4uA/IGUtLDX3g50JH0sMPf9e MNUa9uUW/hKz7iJbU0zOzLOl/zU2VZeyofD7W99CMLPHI1iKGrA1iMdnkGcmDXd/BToo s3p4xYMOEA3jz65+/qIv9llP2GR7KuS7BRga3o0mZ9pxZSvUAyZQGkK0RDUXGGqfUJro ylKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686275110; x=1688867110; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6V2+W0bna42qnZ43tvOf68yFS9Jt1uoq2b6WWLaZ+Rg=; b=JjlTPIjFySg8nOtoBfjWJW8pyak5nuBP73ROObo+Uo8MlgK1eRLz82ZrZVdtZdwdwU DrwhUkKrWKrtwaTwEsJh6XSciNMndBEdTQDPx31izwgq5l5WrctwHKxQ31HuhZUBLazK 0qrkNm6W9GQI1fn8gK55P1ogIyw9BwTPNG90Xv0wi0AWRv/BrWb1tisVskEGFdUido+t DzBZqnVG4ppK7Jf73pXa1LTMMZqkOIex6xNGuryNJpSAV+72TwHIQqS+zeHwcrEYj25d HLF60/UWNitvfAWbg0xCSOzvjX0AC85p3zdQEOkCMT4JLABb9MuhV61U3EXRMzt6FIHY r6vw== X-Gm-Message-State: AC+VfDwCXcV83F7qhzlG3CDwIT0Sf/lJ3uOOLMIglzr0uLo1DZFDwf5D Lh1BA/29HlGz9YsRTn2W3HQg9w== X-Google-Smtp-Source: ACHHUZ6uvLIOAAtIvmaYvvIxcZjU5faumO4qT6eMLe7uSTWFP55mTra/k3W0VJ0vPQcKSj3TeQZVPw== X-Received: by 2002:a81:8884:0:b0:569:74f3:fd07 with SMTP id y126-20020a818884000000b0056974f3fd07mr12140ywf.0.1686275109703; Thu, 08 Jun 2023 18:45:09 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id d63-20020a816842000000b0055a7ff0a5cdsm303405ywc.27.2023.06.08.18.45.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:45:08 -0700 (PDT) Date: Thu, 8 Jun 2023 18:45:05 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 29/32] mm/memory: handle_pte_fault() use pte_offset_map_nolock() In-Reply-To: Message-ID: References: MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: AD2D1A0009 X-Stat-Signature: igugqnpcctndzc37ngb9fhgiyioporu3 X-HE-Tag: 1686275110-549708 X-HE-Meta: U2FsdGVkX1/vpPX1c+Ka+Ze7xuixugEtNxTl+ZWArUYHytgUPzo+0SiFXNEmv6aAuZGHqqz+g+uk/bvfVqQQJK6G5FNw6rcv1iEzIgN/gzVWJ5Ca3rTggjq3+g0+cyXS7cHzy2JdkdT2dmZK09pZ4IZb+8hCXzcmDwysl25i2VQa6mn54aTCK4OwNnvNrtKZ1fQM31eXjvXjVmx+1m66XVuxKlqmDrlnu3yqhgatg+p9PK2JZSIqfAGUQcgkdXc/MRfK8gcZK57tEcpsjNfsGxCBOHcZ3c7cffFlu2wCyVjEFQlrmjH/35bkveXo4hw3oTpEicvXIOF0pNsC1vC9E6pOKgiNo8tn1ox0chTuf2CJcCk+K+Ec+1uCkZor+ENatVCo+jc7P20MTreu5zxdH50KMeFNhsMxWC07uNgZkNp1Z891jT4OErGksW9bQaifNxiEc3i1LvhqYW3BgE4XpkYY5CwkMCnRu0SOISCHKvZPFjsb6k5FLRCVcK6x2D+0CMuS+QttZJopi5PJKzhlUJ/Sit+yG0toXZe/SzKuImgQec6uunBIxfdwO44nDf12iJdPxRSQGgUXrWe4kt4S2YcKzHqtJzQ4JazdbPTOQyCjHS05EHdPI+fLJ10fzWJVnMXuzhAQmVhOw8PrwJyYEFemlMdwsSQ9a2my09sy4bPkJaLF9T1uSXu2IpOt/For4S212wNrZ7EmXZe2j9xCWSf9Gjd0KVThlQJa4uuF93vn8vik86VKoaiOMiWXeFkULLbA0IL0ObikVxDAZlsvM8gIepUZ9dmDR7ucLCdN/EOMiDH4WOsemZzktytXQQp5wd5a8pHIlgl/w6O2eX/NBm1v1WOaA6ddMZ2QH3qhTO+JjkkXO8hbcmHH12/RJlKIvANPRM1PEM4BiuwoOL0jAdluzfas7OASgbq3kkcVKki8KvLe8USfn+99jaM7D4kmKteb7G9kJQmv7w3hR5K dBlT+viR BYqHQHoq4ZmN9aCS2lXXUl7qFd0isBb+MWu+pT3VEHAU6HYAk7KDDqIEcMJreelXLymP1ZqNw2Uy1L0ziv5Nn7e6LoLvrFajpLSzF2e8mNNoa5IbP88C492L2gynYbMpTUk60VQSiB79qOSPTvEdrRhAosEEeb90GAWnZt4pmDiJ7/OVysSqDqVWWXRqCBypk8X6xW4gabTMp+zWBlj1hSba5P0Nc+C0mWmfvyX9ly6UgZ7DxGXDsnb3Ms9IARmp/l8EwT0FSW7wnspzCy4GC3a3xvDL1DKSMIxwLvZga2xabZHrgikFui+v24LdL2+FHbECNGcOf3LmZ5uhdYNKkzWMwZdKqkeaaudycYkkX5c3ISP8JrpbIRl12Bb7Ln7plYReF9AM9++eRIDtB9oNq9S6jNk8NeHQvZoNJgDvNufcufwNmeB1UqYpn3Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: handle_pte_fault() use pte_offset_map_nolock() to get the vmf.ptl which corresponds to vmf.pte, instead of pte_lockptr() being used later, when there's a chance that the pmd entry might have changed, perhaps to none, or to a huge pmd, with no split ptlock in its struct page. Remove its pmd_devmap_trans_unstable() call: pte_offset_map_nolock() will handle that case by failing. Update the "morph" comment above, looking forward to when shmem or file collapse to THP may not take mmap_lock for write (or not at all). do_numa_page() use the vmf->ptl from handle_pte_fault() at first, but refresh it when refreshing vmf->pte. do_swap_page()'s pte_unmap_same() (the thing that takes ptl to verify a two-part PAE orig_pte) use the vmf->ptl from handle_pte_fault() too; but do_swap_page() is also used by anon THP's __collapse_huge_page_swapin(), so adjust that to set vmf->ptl by pte_offset_map_nolock(). Signed-off-by: Hugh Dickins --- mm/khugepaged.c | 6 ++++-- mm/memory.c | 38 +++++++++++++------------------------- 2 files changed, 17 insertions(+), 27 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 49cfa7cdfe93..c11db2e78e95 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1005,6 +1005,7 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm, unsigned long address, end = haddr + (HPAGE_PMD_NR * PAGE_SIZE); int result; pte_t *pte = NULL; + spinlock_t *ptl; for (address = haddr; address < end; address += PAGE_SIZE) { struct vm_fault vmf = { @@ -1016,7 +1017,7 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm, }; if (!pte++) { - pte = pte_offset_map(pmd, address); + pte = pte_offset_map_nolock(mm, pmd, address, &ptl); if (!pte) { mmap_read_unlock(mm); result = SCAN_PMD_NULL; @@ -1024,11 +1025,12 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm, } } - vmf.orig_pte = *pte; + vmf.orig_pte = ptep_get_lockless(pte); if (!is_swap_pte(vmf.orig_pte)) continue; vmf.pte = pte; + vmf.ptl = ptl; ret = do_swap_page(&vmf); /* Which unmaps pte (after perhaps re-checking the entry) */ pte = NULL; diff --git a/mm/memory.c b/mm/memory.c index c7b920291a72..4ec46eecefd3 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2786,10 +2786,9 @@ static inline int pte_unmap_same(struct vm_fault *vmf) int same = 1; #if defined(CONFIG_SMP) || defined(CONFIG_PREEMPTION) if (sizeof(pte_t) > sizeof(unsigned long)) { - spinlock_t *ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd); - spin_lock(ptl); + spin_lock(vmf->ptl); same = pte_same(*vmf->pte, vmf->orig_pte); - spin_unlock(ptl); + spin_unlock(vmf->ptl); } #endif pte_unmap(vmf->pte); @@ -4696,7 +4695,6 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) * validation through pte_unmap_same(). It's of NUMA type but * the pfn may be screwed if the read is non atomic. */ - vmf->ptl = pte_lockptr(vma->vm_mm, vmf->pmd); spin_lock(vmf->ptl); if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte))) { pte_unmap_unlock(vmf->pte, vmf->ptl); @@ -4767,8 +4765,10 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) flags |= TNF_MIGRATED; } else { flags |= TNF_MIGRATE_FAIL; - vmf->pte = pte_offset_map(vmf->pmd, vmf->address); - spin_lock(vmf->ptl); + vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, + vmf->address, &vmf->ptl); + if (unlikely(!vmf->pte)) + goto out; if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte))) { pte_unmap_unlock(vmf->pte, vmf->ptl); goto out; @@ -4897,27 +4897,16 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) vmf->pte = NULL; vmf->flags &= ~FAULT_FLAG_ORIG_PTE_VALID; } else { - /* - * If a huge pmd materialized under us just retry later. Use - * pmd_trans_unstable() via pmd_devmap_trans_unstable() instead - * of pmd_trans_huge() to ensure the pmd didn't become - * pmd_trans_huge under us and then back to pmd_none, as a - * result of MADV_DONTNEED running immediately after a huge pmd - * fault in a different thread of this mm, in turn leading to a - * misleading pmd_trans_huge() retval. All we have to ensure is - * that it is a regular pmd that we can walk with - * pte_offset_map() and we can do that through an atomic read - * in C, which is what pmd_trans_unstable() provides. - */ - if (pmd_devmap_trans_unstable(vmf->pmd)) - return 0; /* * A regular pmd is established and it can't morph into a huge - * pmd from under us anymore at this point because we hold the - * mmap_lock read mode and khugepaged takes it in write mode. - * So now it's safe to run pte_offset_map(). + * pmd by anon khugepaged, since that takes mmap_lock in write + * mode; but shmem or file collapse to THP could still morph + * it into a huge pmd: just retry later if so. */ - vmf->pte = pte_offset_map(vmf->pmd, vmf->address); + vmf->pte = pte_offset_map_nolock(vmf->vma->vm_mm, vmf->pmd, + vmf->address, &vmf->ptl); + if (unlikely(!vmf->pte)) + return 0; vmf->orig_pte = ptep_get_lockless(vmf->pte); vmf->flags |= FAULT_FLAG_ORIG_PTE_VALID; @@ -4936,7 +4925,6 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) if (pte_protnone(vmf->orig_pte) && vma_is_accessible(vmf->vma)) return do_numa_page(vmf); - vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd); spin_lock(vmf->ptl); entry = vmf->orig_pte; if (unlikely(!pte_same(*vmf->pte, entry))) { From patchwork Fri Jun 9 01:50:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273099 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 928FAC7EE23 for ; Fri, 9 Jun 2023 01:50:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F07178E0002; Thu, 8 Jun 2023 21:50:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EB7488E0001; Thu, 8 Jun 2023 21:50:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DA6DA8E0002; Thu, 8 Jun 2023 21:50:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id CB7D28E0001 for ; Thu, 8 Jun 2023 21:50:51 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 9EF3CA03B4 for ; Fri, 9 Jun 2023 01:50:51 +0000 (UTC) X-FDA: 80881530702.15.A3517FB Received: from mail-yb1-f181.google.com (mail-yb1-f181.google.com [209.85.219.181]) by imf10.hostedemail.com (Postfix) with ESMTP id C9F9BC0018 for ; Fri, 9 Jun 2023 01:50:49 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=VYBwsG1B; spf=pass (imf10.hostedemail.com: domain of hughd@google.com designates 209.85.219.181 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686275449; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5QGRbISmNdiSrcRCSB3pJg16AgA7v+5ZDTS/B09Kn8o=; b=w3sYshKbBjeeXTXHoiMd3WsDXr1/q2LBiWx0f1eVqABg2X/Vscuto/Jd154YxqV7Ij7RDH rqwRhGYILytW6aCuLXVmoDildIebmif05B8Ut/TPP2Ru2Ea6cD0OvbP6tNN8g29LiOcwJg Da3ly6SYXLyRiiAAxvN3RdzbYQ+KNHo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686275449; a=rsa-sha256; cv=none; b=ZbuMAFH4hF3uYagu3ZEnts30WU9j9yLqyPLJj3yyxe2szk5ZK6B/7E6LGsUmXGzy+CUC3d SYJZkG6zFjxQY9Z5rxPX/RfksTyCIpzh1xuv70bqXj06VZGXd5G1ia3Cq3uGYM+BW7/bYK 4g6Qn0UehIPtB+oC/D8ByKSD2ilmID4= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=VYBwsG1B; spf=pass (imf10.hostedemail.com: domain of hughd@google.com designates 209.85.219.181 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f181.google.com with SMTP id 3f1490d57ef6-ba829e17aacso1383468276.0 for ; Thu, 08 Jun 2023 18:50:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686275449; x=1688867449; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=5QGRbISmNdiSrcRCSB3pJg16AgA7v+5ZDTS/B09Kn8o=; b=VYBwsG1BtIkMXYG/EqMPmNqaJtObBxLa0DzVTT5ka0MvcTVzYnyBd4s72c34jeMJWZ xJedOpJC62Y06znAOr56dfhffB75a2S0AEDjwGdBtR8M50W2ZRRdIZBxB+7WBisOZrKT nPmJFCweF3RKUaSLr026wSZQ2QVhzWfX9ltBG3Uq6D/Lj/QnQ1Y+MBLl1GEWPRjympq5 k7E6IjPRJAuuYRzzFwPhEP7q5GuBMvHkMBcEHG6JFmqkrVIuIbku6yQIyNpLr2Fx94UY CJ6aDp4I29QQMn44W1tjU0XjRP5XR9NluCQv+G6gcGOq9aMokdcjYhjrO1zsvgIIBXRP l9eQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686275449; x=1688867449; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=5QGRbISmNdiSrcRCSB3pJg16AgA7v+5ZDTS/B09Kn8o=; b=eWQQ+snhjCcT3xsBu861/S9MwQ+hEeLeuzEuruHj3IN+lutxeerzdTng1VzpUAc3cu +MiybAKVbLNfK62d6OuUgw5Z9LaBGLgz4uzkRyaUvWn5hif9jA61FsMoQvpaud+BmOam umIc+JpeQxAQq1FB5nFn9kwurI4WShzpguNlA5ybQJdYDQ2FgFalxOI/FhkcqqAKJsvZ yK/l2/v6GI+/SUSFmkz+ThhqRw/U7eJhuPdFeC+bQRz2GqEoQliyKGQ6zTZ8oeUHieyf P15QmIEoK8/6cpD0H+Lzj/Bb1D9Kogbkb4OREC6Lz869lyBlz+H11sQh7YAiZLBlutfM KksQ== X-Gm-Message-State: AC+VfDykj9hlfhszS1jj3O12t3Wd03qCPY9RD7XytS8wlh2qZ0HYjSIR ARWcnNPvEnn6He3u4fJFYp/Igw== X-Google-Smtp-Source: ACHHUZ7rkKxLDUXb7gVbThinQJUAuYEpzA2mTjY1b2VjU6hE71QC6+GRYBvyPmvE/ofxxSjMXUupyg== X-Received: by 2002:a25:4103:0:b0:ba8:8779:667 with SMTP id o3-20020a254103000000b00ba887790667mr1159827yba.41.1686275448723; Thu, 08 Jun 2023 18:50:48 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id n11-20020a25400b000000b00bb393903508sm622475yba.14.2023.06.08.18.50.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:50:47 -0700 (PDT) Date: Thu, 8 Jun 2023 18:50:37 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 30/32] mm/pgtable: delete pmd_trans_unstable() and friends In-Reply-To: Message-ID: <5abdab3-3136-b42e-274d-9c6281bfb79@google.com> References: MIME-Version: 1.0 X-Rspamd-Queue-Id: C9F9BC0018 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: x6z8khrbzqtrzc7gmi85stwjiqfgfjxm X-HE-Tag: 1686275449-281327 X-HE-Meta: U2FsdGVkX18crguUBMkPxAyCGDrs4ESfAWKL9h5cYhjXvkm5KPg0zu3OeHrMedY4XCZZl2i5+tSG6bMT8hCOAH9H/h65OhLr9jQCfQjem9LW9Nqf2qogELmZqGT2a7TV4yNk2c/8AbKAkc3Yfw+3hJ1jE/0YhimSDVZA/1nkDBrVkkB5SzSAHmGp41plBXXVv6o/lvoAp3LQ0WK+2reqe9OopfEKOOpiIBzAj/YJvW6Acrph21IZnYB5EhHtJv7hQQcEVgL/J+yyq9L4Aaf6HFYo63vDTem/TwWfilZ7Bw11Fdb/x/RTQdMq55Y3NeIbWby50mng+s7ZdPwrC4ZxkupV7npdjekzL1c5rJxoZr6zefW89TTifJS2GrwCQEuTkAxYc3emSPvBLUCKH8btQpOkbzNGEzE9+VCB32FjNOMjw2gc2g7vqZBedbalnYrTbS33mbRNeNy/svLeafF5Vg4jG1HyQryg/FEA05eCU64P8GduCt6yk+/wVvw78I1WBhwwtBoWwbi5ZQ0lpRagytvmDFKFM4nPzOQGrOlRbdsTLROW6VvNPKhyhOffWc/cuPdghEt1YkK5V/IBGq8fabphurxyX7hLBX9SSYLzRHPNasevPC/8kq0VjJnmKtYZcB4DPNYAso00sAec/RVUAoe8tjypxmsnP31C8uXaboe1fFdGcsPTF0R878+hp06yCYuOMSLwTL5B2YZjdFf/XWqRBNyzxluNkmJX+zB2yl2a3MzwjM3a15ldUYbSzP4H0sx3cTtmfXq4nCyDLjKKFuiEcEX3RncAv7a1hqnUtCEg5Grv0g5s17+jAB+zIMcXhS92yF/g08vbDUPP4+vaEbtXEZt2rbv9eLXC5OZ4EM4xX7M5b22Goxnlk02Uce4MF+XM2wimpUuU7Cfy/hMhpMwycOZLue0Z+gf4iK4SSmuSADjhim/btTjdJ2yR0TljgoWCdR3KJc4xDW+WTCf og/JYu92 VdDd1R+b1NSs5/tM4JfwkbayqmnhCHLuHlXWspN2qQtwZH2x0eGvyZsTDm26zWFDfTEys+3NNGCRGXGU41MkG49z/xkm+q8ffcmgJ1qreV6Tcf57UxalQBPmjylvKD44RIPcC9V1nar9xv177MBz2DWPRsKACTfJH+XvzMZ5/oFnofsZoML70auweEN6KgZnlJ/6hUgi1S77iGQa6wofDMzWgKYgx8wioREolZzqzhXSNoYeJvqlTx2IKOEzI4whSC5doyahAN8LfmiC1dzE50Sqp9nF/hjKQU+u1IfPQEDfTMmd5n8t+6mXvzjjRf5P9LezQjCLi9iz+mA5GYfF7d4zPPeEgFP5jfUA2Q9GYL4AhhJT5O3OIobsSX4aJkFY2sbz4qWndLGqEUf15j6Lxky8tFfOMT6pn82+17mb453GXmWGtXdzrmKZuHg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Delete pmd_trans_unstable, pmd_none_or_trans_huge_or_clear_bad() and pmd_devmap_trans_unstable(), all now unused. With mixed feelings, delete all the comments on pmd_trans_unstable(). That was very good documentation of a subtle state, and this series does not even eliminate that state: but rather, normalizes and extends it, asking pte_offset_map[_lock]() callers to anticipate failure, without regard for whether mmap_read_lock() or mmap_write_lock() is held. Retain pud_trans_unstable(), which has one use in __handle_mm_fault(), but delete its equivalent pud_none_or_trans_huge_or_dev_or_clear_bad(). While there, move the default arch_needs_pgtable_deposit() definition up near where pgtable_trans_huge_deposit() and withdraw() are declared. Signed-off-by: Hugh Dickins --- include/linux/pgtable.h | 103 +++------------------------------------- mm/khugepaged.c | 4 -- 2 files changed, 7 insertions(+), 100 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 3fabbb018557..a1326e61d7ee 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -599,6 +599,10 @@ extern void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp, extern pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp); #endif +#ifndef arch_needs_pgtable_deposit +#define arch_needs_pgtable_deposit() (false) +#endif + #ifdef CONFIG_TRANSPARENT_HUGEPAGE /* * This is an implementation of pmdp_establish() that is only suitable for an @@ -1300,9 +1304,10 @@ static inline int pud_trans_huge(pud_t pud) } #endif -/* See pmd_none_or_trans_huge_or_clear_bad for discussion. */ -static inline int pud_none_or_trans_huge_or_dev_or_clear_bad(pud_t *pud) +static inline int pud_trans_unstable(pud_t *pud) { +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && \ + defined(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD) pud_t pudval = READ_ONCE(*pud); if (pud_none(pudval) || pud_trans_huge(pudval) || pud_devmap(pudval)) @@ -1311,104 +1316,10 @@ static inline int pud_none_or_trans_huge_or_dev_or_clear_bad(pud_t *pud) pud_clear_bad(pud); return 1; } - return 0; -} - -/* See pmd_trans_unstable for discussion. */ -static inline int pud_trans_unstable(pud_t *pud) -{ -#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && \ - defined(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD) - return pud_none_or_trans_huge_or_dev_or_clear_bad(pud); -#else - return 0; #endif -} - -#ifndef arch_needs_pgtable_deposit -#define arch_needs_pgtable_deposit() (false) -#endif -/* - * This function is meant to be used by sites walking pagetables with - * the mmap_lock held in read mode to protect against MADV_DONTNEED and - * transhuge page faults. MADV_DONTNEED can convert a transhuge pmd - * into a null pmd and the transhuge page fault can convert a null pmd - * into an hugepmd or into a regular pmd (if the hugepage allocation - * fails). While holding the mmap_lock in read mode the pmd becomes - * stable and stops changing under us only if it's not null and not a - * transhuge pmd. When those races occurs and this function makes a - * difference vs the standard pmd_none_or_clear_bad, the result is - * undefined so behaving like if the pmd was none is safe (because it - * can return none anyway). The compiler level barrier() is critically - * important to compute the two checks atomically on the same pmdval. - * - * For 32bit kernels with a 64bit large pmd_t this automatically takes - * care of reading the pmd atomically to avoid SMP race conditions - * against pmd_populate() when the mmap_lock is hold for reading by the - * caller (a special atomic read not done by "gcc" as in the generic - * version above, is also needed when THP is disabled because the page - * fault can populate the pmd from under us). - */ -static inline int pmd_none_or_trans_huge_or_clear_bad(pmd_t *pmd) -{ - pmd_t pmdval = pmdp_get_lockless(pmd); - /* - * !pmd_present() checks for pmd migration entries - * - * The complete check uses is_pmd_migration_entry() in linux/swapops.h - * But using that requires moving current function and pmd_trans_unstable() - * to linux/swapops.h to resolve dependency, which is too much code move. - * - * !pmd_present() is equivalent to is_pmd_migration_entry() currently, - * because !pmd_present() pages can only be under migration not swapped - * out. - * - * pmd_none() is preserved for future condition checks on pmd migration - * entries and not confusing with this function name, although it is - * redundant with !pmd_present(). - */ - if (pmd_none(pmdval) || pmd_trans_huge(pmdval) || - (IS_ENABLED(CONFIG_ARCH_ENABLE_THP_MIGRATION) && !pmd_present(pmdval))) - return 1; - if (unlikely(pmd_bad(pmdval))) { - pmd_clear_bad(pmd); - return 1; - } return 0; } -/* - * This is a noop if Transparent Hugepage Support is not built into - * the kernel. Otherwise it is equivalent to - * pmd_none_or_trans_huge_or_clear_bad(), and shall only be called in - * places that already verified the pmd is not none and they want to - * walk ptes while holding the mmap sem in read mode (write mode don't - * need this). If THP is not enabled, the pmd can't go away under the - * code even if MADV_DONTNEED runs, but if THP is enabled we need to - * run a pmd_trans_unstable before walking the ptes after - * split_huge_pmd returns (because it may have run when the pmd become - * null, but then a page fault can map in a THP and not a regular page). - */ -static inline int pmd_trans_unstable(pmd_t *pmd) -{ -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - return pmd_none_or_trans_huge_or_clear_bad(pmd); -#else - return 0; -#endif -} - -/* - * the ordering of these checks is important for pmds with _page_devmap set. - * if we check pmd_trans_unstable() first we will trip the bad_pmd() check - * inside of pmd_none_or_trans_huge_or_clear_bad(). this will end up correctly - * returning 1 but not before it spams dmesg with the pmd_clear_bad() output. - */ -static inline int pmd_devmap_trans_unstable(pmd_t *pmd) -{ - return pmd_devmap(*pmd) || pmd_trans_unstable(pmd); -} - #ifndef CONFIG_NUMA_BALANCING /* * Technically a PTE can be PROTNONE even when not doing NUMA balancing but diff --git a/mm/khugepaged.c b/mm/khugepaged.c index c11db2e78e95..1083f0e38a07 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -946,10 +946,6 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address, return SCAN_SUCCEED; } -/* - * See pmd_trans_unstable() for how the result may change out from - * underneath us, even if we hold mmap_lock in read. - */ static int find_pmd_or_thp_or_none(struct mm_struct *mm, unsigned long address, pmd_t **pmd) From patchwork Fri Jun 9 01:52:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273100 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E27CC7EE23 for ; Fri, 9 Jun 2023 01:52:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A71FC8E0002; Thu, 8 Jun 2023 21:52:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A227E8E0001; Thu, 8 Jun 2023 21:52:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C4BD8E0002; Thu, 8 Jun 2023 21:52:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 7C6AC8E0001 for ; Thu, 8 Jun 2023 21:52:25 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 50FA280380 for ; Fri, 9 Jun 2023 01:52:25 +0000 (UTC) X-FDA: 80881534650.24.32A0133 Received: from mail-yb1-f177.google.com (mail-yb1-f177.google.com [209.85.219.177]) by imf10.hostedemail.com (Postfix) with ESMTP id 754DCC0015 for ; Fri, 9 Jun 2023 01:52:23 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=PYctsQ9b; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of hughd@google.com designates 209.85.219.177 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686275543; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MICzT8g6d2gbosF05gYeCNQyYA3E/PyjzMQZ3kC9KOI=; b=juzIS613ypA/N0ju0hwO2HxwiqebYZUp5aJmms9/6GU3/npXi3gYP1xXXA8Ijmg/rJ3Dk/ 1iZRfqd5zzHSn+MBx2CeA4AJ6ie8dUem7f2qe+YDpon6yZeBt+Wy1IE/kPptWRl0iayDER 6PCf1nE3J8Pxpm/bO00GgbdHk50LN6w= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=PYctsQ9b; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of hughd@google.com designates 209.85.219.177 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686275543; a=rsa-sha256; cv=none; b=IZ66ELnkUnjDPBGxaQbbJBOSUYiHmlIrpsd5bAdwvBfnYDRI4PGOXSU6ZdPHbPHkjmuOO1 5mJZw240EXCIt9fxEGOaZBjK1dXxEeP6pzZkPg8BeVawUu/Sx5jZsEmdClAvfqi4J0t4T5 SAPuXCVoMURUA4cl2CvdDLsCNiFHN54= Received: by mail-yb1-f177.google.com with SMTP id 3f1490d57ef6-ba829e17aacso1384567276.0 for ; Thu, 08 Jun 2023 18:52:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686275542; x=1688867542; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=MICzT8g6d2gbosF05gYeCNQyYA3E/PyjzMQZ3kC9KOI=; b=PYctsQ9b4HrQorKzO7zX+fdJyjXUiN8i4W2q9bIpl7bYnCcy9KOZVUhjq8Som0kotl 8hN3DheRVe/kgHsmckj1HIHUS7QkhDrduKAtxjWrJ7uPd3Z1kPi0K/a74/drhVMrEv5N iyFn0O/U0PF+LJj7jEIWaYX3IwAMgIU4MQFfWd5rKkg5CQiQ4VOa9f2TnjccivpRjNpe MLfjrjWUH5cF/6qTPBYKfnjLb/mQJxgyL/APasFw6ccuLxdD/9qF0oXkbHPENouaIFKL go5KxwidvL12GJkGeTVlF7tb8Sm8kZvi4Yzd/FTjpbEMUsi8U+HBiLJXyA+0bfuLk/Ed ypDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686275542; x=1688867542; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MICzT8g6d2gbosF05gYeCNQyYA3E/PyjzMQZ3kC9KOI=; b=V0g+6yk2bdfqBfFca9n8K/ZPRbY+RK8IgB4WXmTm+Cds3nmaFxq2lXmGxcVLFbqrzS JiXMZrDyL0+Pteo/ufj1lp85H99/K+0rL6vwbHpHfFcUvKURbT0ITDmhtwd1DX39K6Y8 Hi0QWB6jKExcsmpC9HkF4dYEjQ8jg87CyaXVoM3azxrCxlrqHa3eguDd7M/6p6e3wNXj LQrd8lgOwEWwoyV0ZYTnuPFtmQMCgjYdsfSShkCunP4GusOKFpJEBRPc4lpSbwE8U6t8 FMFY3I4W+G0mPqqHZjYfeHx8bG366Wa4MiiBjUDOGhwVzC0zOYreQuTam44LA6F8xZCk fnjg== X-Gm-Message-State: AC+VfDzb6WNxCwAO9Y0C+epJwcOqfQ4sOuSworXjLJsZ/lROPza6IaET B9lCKnUq/9UTaT6EK1G7Jv21+g== X-Google-Smtp-Source: ACHHUZ4s2iHF1O1eWsJO+RR6O7kpy/Z7YTwwVNBnE8hanvEwdCoqR2iged2HkuPKJixJaGakJ6Kn7w== X-Received: by 2002:a0d:cf86:0:b0:559:ed0a:96c4 with SMTP id r128-20020a0dcf86000000b00559ed0a96c4mr1168635ywd.44.1686275542179; Thu, 08 Jun 2023 18:52:22 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id j81-20020a816e54000000b00569e7cbcd56sm300407ywc.69.2023.06.08.18.52.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:52:21 -0700 (PDT) Date: Thu, 8 Jun 2023 18:52:17 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 31/32] mm/swap: swap_vma_readahead() do the pte_offset_map() In-Reply-To: Message-ID: References: MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 754DCC0015 X-Stat-Signature: 45n7r1gbpnwc5mmgqeja77xnxdm6u8kb X-Rspam-User: X-HE-Tag: 1686275543-579074 X-HE-Meta: U2FsdGVkX19N1G86UaswR3/Tlnv58Y8CZO2w5o1wJd4z0HSEbcwRcSJBljcDEqA9WimfnSX9156eFY8MzAdsbUYxrz9J8dVYRAiDVjPneJoij6PK1SrMIKs9UBeRvePi+Et+MuPan6hN5/zz+iJhF+03XAZ2qbstBs6Yekz6V/6qG/g/ZfawGPW/YP51diziiGasDcn6a52zdbokTkT8MmVwMzcofXrz54WdwHrboINjp9yhNMzAqyjGRzm9JCeGDFQFRKgLg1OkchVUSYLgCQmylqNj5QgwaYwyVuYErwApyJe9Rhl5Az3v2yrmIWOi9129HWLTaiuU1a+QDTVvcVoXZcFz6kqrGpV2xuA2YyM1w8PiumgZbgsRgaah8Izu8Ri4usjtAh5UCDC5qes2Cg3i7aExm5UnfuAEmppyJEX+9MhO+okG0x4s84b37ViZ7u5Na7cTP6SoPvv50HFhGziMezr07zwVXmvZHQ38qzVmYhZFO71jcZCDauryjj+hNB5gcV6MRM9WxRxi150u1toHsi50fXJlnSVB9hHt9bQl4yr2QcK8kb+vApv6rSbqq95Jec4cTfF+kobOTbc5L6g9ci117UJdEorjJ4srPoEorqdn4ein0g8iZGP02PZX3y3GRHQ35nwTaRu3y4EAnWHED9HpDmIdOHFltdsqHtP0qWIw/LOn2Sz8y3xRVAUDXqCy73Lx+wAkW+sALkZY8USFcpSscSjm41Xx2vMMD7udYP+TWwJ4q5nULivR5+E/GtsJlFxxo/o3FudWvx/oTdgnIuE7QVSv1NlvUhv/ZvE+IWx6bid6c4iB1fBw7qIPyMlPVX+YxFGYmrj3WQqOIO+in6uy6gW3qVAR5Ks2aQ4mOyvYKJ7yIkmx1pRw1zwlelM6L72FZ0T+50PBQu53yej3OKFRqofNR7n8/jTVKNVLsc3iFGAndAFFg653CkamQ+khgOvYEtqSYQfLiTA Fh3xu18W sT0xnuCt91OtNLh1QA8g/j5EQ5KMAOKQj9OIgjAHdQMMrQfjkL2bga0A0WVW4a/yEBX/tMT+opgTFjmfkLBsMvM2dpqpjO3/OUyjQ64azutbf0QJbCRznVFd/b9mRBNLZ1CkHUjAtlDAo+u1GALWASAjOfM6fheHr3YhPhu+kKS0eGpnmjeVJOwMMN3YF4cWbvTkS3CYH87jfdOHWHq1bIuT/gZGzEn+5SUab1Mh9Aa7k0Ew+CJAyaR+/WFJrMuWo2Dl6UUBOg8vnjL91HuUpsL1sWTvrV2AzNZWvLwHDBGOaLG5td5j8mwnKr2w170MV8GxkxMP23pa38iPaYNRvcCJwGjkZ+kAQJXj+pboRRU+i2RzplP1BxIX7dN+fDIUQeUPc/AqJ00kuFlXLwsegj97x0TxX13MJ5vG4+yK6yNaSkRPD0hih4zNV9A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: swap_vma_readahead() has been proceeding in an unconventional way, its preliminary swap_ra_info() doing the pte_offset_map() and pte_unmap(), then relying on that pte pointer even after the pte_unmap() - in its CONFIG_64BIT case (I think !CONFIG_HIGHPTE was intended; whereas 32-bit copied ptes to stack while they were mapped, but had to limit how many). Though it would be difficult to construct a failing testcase, accessing page table after pte_unmap() will become bad practice, even on 64-bit: an rcu_read_unlock() in pte_unmap() will allow page table to be freed. Move relevant definitions from include/linux/swap.h to mm/swap_state.c, nothing else used them. Delete the CONFIG_64BIT distinction and buffer, delete all reference to ptes from swap_ra_info(), use pte_offset_map() repeatedly in swap_vma_readahead(), breaking from the loop if it fails. (Will the repeated "map" and "unmap" show up as a slowdown anywhere? If so, maybe modify __read_swap_cache_async() to do the pte_unmap() only when it does not find the page already in the swapcache.) Use ptep_get_lockless(), mainly for its READ_ONCE(). Correctly advance the address passed down to each call of __read__swap_cache_async(). Signed-off-by: Hugh Dickins Reviewed-by: "Huang, Ying" --- include/linux/swap.h | 19 ------------------- mm/swap_state.c | 45 +++++++++++++++++++++++--------------------- 2 files changed, 24 insertions(+), 40 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 3c69cb653cb9..1b9f2d92fc10 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -337,25 +337,6 @@ struct swap_info_struct { */ }; -#ifdef CONFIG_64BIT -#define SWAP_RA_ORDER_CEILING 5 -#else -/* Avoid stack overflow, because we need to save part of page table */ -#define SWAP_RA_ORDER_CEILING 3 -#define SWAP_RA_PTE_CACHE_SIZE (1 << SWAP_RA_ORDER_CEILING) -#endif - -struct vma_swap_readahead { - unsigned short win; - unsigned short offset; - unsigned short nr_pte; -#ifdef CONFIG_64BIT - pte_t *ptes; -#else - pte_t ptes[SWAP_RA_PTE_CACHE_SIZE]; -#endif -}; - static inline swp_entry_t folio_swap_entry(struct folio *folio) { swp_entry_t entry = { .val = page_private(&folio->page) }; diff --git a/mm/swap_state.c b/mm/swap_state.c index b76a65ac28b3..a43b41975da2 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -698,6 +698,14 @@ void exit_swap_address_space(unsigned int type) swapper_spaces[type] = NULL; } +#define SWAP_RA_ORDER_CEILING 5 + +struct vma_swap_readahead { + unsigned short win; + unsigned short offset; + unsigned short nr_pte; +}; + static void swap_ra_info(struct vm_fault *vmf, struct vma_swap_readahead *ra_info) { @@ -705,11 +713,7 @@ static void swap_ra_info(struct vm_fault *vmf, unsigned long ra_val; unsigned long faddr, pfn, fpfn, lpfn, rpfn; unsigned long start, end; - pte_t *pte, *orig_pte; unsigned int max_win, hits, prev_win, win; -#ifndef CONFIG_64BIT - pte_t *tpte; -#endif max_win = 1 << min_t(unsigned int, READ_ONCE(page_cluster), SWAP_RA_ORDER_CEILING); @@ -728,12 +732,9 @@ static void swap_ra_info(struct vm_fault *vmf, max_win, prev_win); atomic_long_set(&vma->swap_readahead_info, SWAP_RA_VAL(faddr, win, 0)); - if (win == 1) return; - /* Copy the PTEs because the page table may be unmapped */ - orig_pte = pte = pte_offset_map(vmf->pmd, faddr); if (fpfn == pfn + 1) { lpfn = fpfn; rpfn = fpfn + win; @@ -753,15 +754,6 @@ static void swap_ra_info(struct vm_fault *vmf, ra_info->nr_pte = end - start; ra_info->offset = fpfn - start; - pte -= ra_info->offset; -#ifdef CONFIG_64BIT - ra_info->ptes = pte; -#else - tpte = ra_info->ptes; - for (pfn = start; pfn != end; pfn++) - *tpte++ = *pte++; -#endif - pte_unmap(orig_pte); } /** @@ -785,7 +777,8 @@ static struct page *swap_vma_readahead(swp_entry_t fentry, gfp_t gfp_mask, struct swap_iocb *splug = NULL; struct vm_area_struct *vma = vmf->vma; struct page *page; - pte_t *pte, pentry; + pte_t *pte = NULL, pentry; + unsigned long addr; swp_entry_t entry; unsigned int i; bool page_allocated; @@ -797,17 +790,25 @@ static struct page *swap_vma_readahead(swp_entry_t fentry, gfp_t gfp_mask, if (ra_info.win == 1) goto skip; + addr = vmf->address - (ra_info.offset * PAGE_SIZE); + blk_start_plug(&plug); - for (i = 0, pte = ra_info.ptes; i < ra_info.nr_pte; - i++, pte++) { - pentry = *pte; + for (i = 0; i < ra_info.nr_pte; i++, addr += PAGE_SIZE) { + if (!pte++) { + pte = pte_offset_map(vmf->pmd, addr); + if (!pte) + break; + } + pentry = ptep_get_lockless(pte); if (!is_swap_pte(pentry)) continue; entry = pte_to_swp_entry(pentry); if (unlikely(non_swap_entry(entry))) continue; + pte_unmap(pte); + pte = NULL; page = __read_swap_cache_async(entry, gfp_mask, vma, - vmf->address, &page_allocated); + addr, &page_allocated); if (!page) continue; if (page_allocated) { @@ -819,6 +820,8 @@ static struct page *swap_vma_readahead(swp_entry_t fentry, gfp_t gfp_mask, } put_page(page); } + if (pte) + pte_unmap(pte); blk_finish_plug(&plug); swap_read_unplug(splug); lru_add_drain(); From patchwork Fri Jun 9 01:53:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13273101 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 968D9C7EE29 for ; Fri, 9 Jun 2023 01:53:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 394528E0003; Thu, 8 Jun 2023 21:53:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 344048E0001; Thu, 8 Jun 2023 21:53:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 20C2B8E0003; Thu, 8 Jun 2023 21:53:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 12E568E0001 for ; Thu, 8 Jun 2023 21:53:31 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id E32C81A039B for ; Fri, 9 Jun 2023 01:53:30 +0000 (UTC) X-FDA: 80881537380.24.043D669 Received: from mail-yw1-f173.google.com (mail-yw1-f173.google.com [209.85.128.173]) by imf10.hostedemail.com (Postfix) with ESMTP id 1B086C0002 for ; Fri, 9 Jun 2023 01:53:28 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=W1vFMPti; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of hughd@google.com designates 209.85.128.173 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686275609; a=rsa-sha256; cv=none; b=jBCPQw9ATXV+35VOBraArPYFiCq/5IM9JGXUuYXJ3x0TreYGRNE4xxv3lhz9Bwg8FHmD7f k49rR+fJzR5yZcz00UBAZdM920NLUOjeqjuVlDBgHLBZXdMNBH6wps672EVFfCsE3G3EL1 8yvsiXvCSiKR2/8tuW/q0ssJTPogTGU= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=W1vFMPti; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of hughd@google.com designates 209.85.128.173 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686275609; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=++qSmbn7bhjg3RT15iglmwv4R1gESGfbvlyrc9z9nPg=; b=NC1Qf8Q+Io6yqPRvDsogEZjQkUEyqrvwDusHC+4lBjimv+LYnpgi7Hr3ZqFbOSXnJNFO6M s6PvBGaHJlCsEWY2Az5DzgydwWfTBLhU1C+yC2tBtxFmZ6GN4kp1m7wukS5p8ZM1DTqbtf OKyyZDMlMAs+I3uZ0xXaM3WvCvKkNQ0= Received: by mail-yw1-f173.google.com with SMTP id 00721157ae682-568900c331aso12296867b3.3 for ; Thu, 08 Jun 2023 18:53:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686275608; x=1688867608; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=++qSmbn7bhjg3RT15iglmwv4R1gESGfbvlyrc9z9nPg=; b=W1vFMPtinSndBcBgNlPbjuG1vI+MS4h79C73H3xECr5w/wVd+ID5QFfuBCbZ7w3Ayy CqgKuvbsR/UytKoF7Wy8rKKREq+DfXmPMzwuIXXWHCfXjS6w2FOZgMYNRgBe70zqe7iJ hE/jG7WdGUizvK/G6fx0GyFk9oLfadtxrkg+RkvFJ+RmYzOn5qIxJuKlmaDNTeDZEEE4 Q9LEWq0qVGafHnb3ONnwEEktOqfHdhe87GaAA+HqXKGM3CI972T54tFojES5P7841qpO GKVXJvYG8HirDB7MzuADX+/RcwH8/w5ThO2DEpKlo8tc7BvvsNgYXTbXSVtPO7qfqPxu lwNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686275608; x=1688867608; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=++qSmbn7bhjg3RT15iglmwv4R1gESGfbvlyrc9z9nPg=; b=WRP+Uze24VZcOnZuMW1sD/4SMaTWMW/RtDb+saor+lOwrGMdkLFFKS9m1RlPrJZHvM wUpTdKbS8csDg9/0rJCBTM/mHdxwC4HuZZWNGLrLpoVoEUAx+/WiI7J8W8hwlKDU2anH mAXoTG2MzTO/JI24RPCLmmUOVydaV0iwLBRMop2BFs8SystOJTCKr5ybBv2m2qJbBzNS rO9BoJCvM1kehAyeS2GvSs17ZaO3/4rDf9K4lRP1XqWpbJOXpYj3ao9Bax+pHkV5T4Tl 1DV/iJPOo2oDn56xkPw3L8mjKpb7/FU2Ja4lrfVPEHBmkRzDj0QFDrHLY1dRjKERt0S8 bJrQ== X-Gm-Message-State: AC+VfDz3/SKe0V7PnnXpUrwLqQxiPpfmxHsqLvOjMlC+Uf1kS3k48RoB wcoyqetcTTxOPJH8YH6GHMwZlA== X-Google-Smtp-Source: ACHHUZ7ms7OAKGCZkRczhA1SWfMIL/NPHAZni0NvY//Wj9lDtLQhlFsfX2EowXAYpVWs6Ek/Wh+UAA== X-Received: by 2002:a81:8403:0:b0:565:eae8:793b with SMTP id u3-20020a818403000000b00565eae8793bmr1235947ywf.25.1686275608094; Thu, 08 Jun 2023 18:53:28 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id r125-20020a0de883000000b00568c29c3c4csm307485ywe.38.2023.06.08.18.53.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:53:27 -0700 (PDT) Date: Thu, 8 Jun 2023 18:53:23 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 32/32] perf/core: Allow pte_offset_map() to fail In-Reply-To: Message-ID: References: MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 1B086C0002 X-Stat-Signature: 89ccfyya8kfe5r6xqy97fgbin3rxd9b3 X-HE-Tag: 1686275608-468374 X-HE-Meta: U2FsdGVkX1/QSXpsvxpwMGwC+nXNYV+R9DDHJKGrNFNaR/ST2JhS32+lCSqoaq3i6o2aedvZhiGTj88Z4mgzE+wSKZ2o4DVk8RTZol85mxKY4HICGyv/+YEybE/LjYN4BkviimqG0jHQ9dzwwLh55gsbBlLtd3wwlduFRjtum5u0lTp3aox6gtKekrlHXm7ksM1wa7n6GRyrVmbCotir/B3TYEuQb4ncZA2tXZZic1tv3KoqAAtf78CuxxeSrUelSwXf+Uefs1YKv4V9V8T7gMezFIuOHFfAw+DJKscK0yst/3bkqEo+/awxpnC3yZ2mj+B8wOHzO76UpgCYw7DYhGW8rgHplrUlS+pnhi4om7DlD88lyB3fOPipvp1JW8nnOEny3qI/1IyWzF/EybZ2pCcryTUcjnoXoXkiXggSmUgwWy7ykrGGop5JdV57MMeZa2LsAp+cW4yyI803sRm90NN/xCPuESrL9xNHwktvD1YmGnVE/TdDRcBEdCBaFESvsTWrqfT3ewMzvxgJ4bmDWmh0vr0FcvsRJ1s6vm2Oic/HQtxvUV9jdRy62pp8ZDanIs24ekdsW0QeY5y3rF5xED9elGoTRa3yEiVzTWnmUN1WsVesedL0jBGIAfEK5EX6nWPYwMO6sxRLBk2U77RHATv8FRSGQ+k/Y23Na0sqLEcb1ZuoWnEMj8sjTDp057vNJ3Hf4A8X4NWf1L8kXuNlgmFutr/uMw3VBR8b42nKO3Rjp8x+RLs+9vqiMwdDo30efJf8f1BpiAJkbZvFbKxwvmTGD2EImJxkHM3wmr6xvrUge0/yAj/Lx4zoSlipnfeQrnXhRgF+W+6476qvNJ5lOZH4TgGV66rZX+NOru9XcusTHvFwjaHOaYEXgfxDyKOqUYy3tjZFzWKawtXvrHFd3WiABDWNdGiGwOkJVk4gILgUdy+aCK7MMwxD4zTLBYpkIYFlT0m+FBE3kHJy3CW XB86kI0O DXx2CTn/s3tkoKv9j0SuEH8T2nE2Zsj0TJItxLS0Y3NY6wcVv0YWNZwVJ0921VWep7+oSI0fM3sLMzRtb5IDO/ScfvV370w2lEAjfDZ7I/7NthpmvlQmW7Pdf2RtiMSHSw2ocn6hKGGFncFEPOKSgV6KOQ84Yod9vriAKLNoDdP6sew74eaL6qlOc7TQVSCOob+rDhLWCLAUG766Riebo3YxN62vRRAcBQqf5+1XtPAwicDA4xldWlkT4FhC7mydFoGPY7zwsrhbZctT/HWTRQxYULoHNmI0BM1tjoqFP1l6bZdO2TF/zw3u4Z9U/08PnLcXtf1WKfN64wWybnSpfXsRdbQhvq/0LYcHfoqu8zPwQiXG8HjPGFUYFaKX9dvEimI1ArxiIWrYVRSZZCprmh9KDk3bRgvpoGpm5E3vNrGOln6GAzvhAYXrEQw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In rare transient cases, not yet made possible, pte_offset_map() and pte_offet_map_lock() may not find a page table: handle appropriately. Signed-off-by: Hugh Dickins --- This is a perf patch, not an mm patch, and it will want to go in through the tip tree in due course; but keep it in this series for now, so that it's not missed, and not submitted before mm review. kernel/events/core.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/kernel/events/core.c b/kernel/events/core.c index db016e418931..174be710f3b3 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -7490,6 +7490,7 @@ static u64 perf_get_pgtable_size(struct mm_struct *mm, unsigned long addr) return pud_leaf_size(pud); pmdp = pmd_offset_lockless(pudp, pud, addr); +again: pmd = pmdp_get_lockless(pmdp); if (!pmd_present(pmd)) return 0; @@ -7498,6 +7499,9 @@ static u64 perf_get_pgtable_size(struct mm_struct *mm, unsigned long addr) return pmd_leaf_size(pmd); ptep = pte_offset_map(&pmd, addr); + if (!ptep) + goto again; + pte = ptep_get_lockless(ptep); if (pte_present(pte)) size = pte_leaf_size(pte);