From patchwork Wed Sep 26 21:08:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 10616833 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1F029174A for ; Wed, 26 Sep 2018 21:09:06 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0F2D52B774 for ; Wed, 26 Sep 2018 21:09:06 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 038B32B7F9; Wed, 26 Sep 2018 21:09:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 386E42B774 for ; Wed, 26 Sep 2018 21:09:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727097AbeI0DXx (ORCPT ); Wed, 26 Sep 2018 23:23:53 -0400 Received: from mail-qt1-f196.google.com ([209.85.160.196]:34032 "EHLO mail-qt1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726463AbeI0DXw (ORCPT ); Wed, 26 Sep 2018 23:23:52 -0400 Received: by mail-qt1-f196.google.com with SMTP id x23-v6so531098qtr.1 for ; Wed, 26 Sep 2018 14:09:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references; bh=epHd1o6KPupVYmL3VXxvF5Z1JP8sYlPDk5pYju44IUA=; b=C/WmY+fOpp8biQhBYCH31rM3jxFDT8U3+B6k3+P9k8a8uUBjjm3DsNlaVszOpkOIQo MhoaXlP5edZvLpW2AIIau1epngcB3LJW6tcdkvtv62DhvqCZYUDfiMhQ3ztyWn6xckA0 f3tSc1ecKkFzgWbjR2rjl/LEhRkA3YO6uO6jkD8l4Kd72aS1NsJu2cVUB854i7pFgAEQ HkfYEKZvYHwb2g4McN7JDAzlCZ5Khe/LC4AKkNCgcsoVBZHN0iR13srqaEyZdObdP2W1 GWHBvX00aDWXr2TmSdd1W8h4iWKF8Lx93fIMGJksmT2EbeBONfqSXG8ZFqC/OZk/utfH bV2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=epHd1o6KPupVYmL3VXxvF5Z1JP8sYlPDk5pYju44IUA=; b=GoYTYqDw0K1bg2fTN+Wuzs4KF8BgvjVYP+1LaMWWn30jgUrXssQ9R9oXss9BdHbF90 lnuT2TI5TQ0pOpHKgaMfj2CA0KqNHOzdCcZsSV/bMPLx08RfOpqqd+YrUVQekW0ow+Zb CNQI2ys9mDbyPlUrKF5ikvnjzAI5ZYWwNrmOL8Eu62XcVfC7ImyvO4MszrdaCckXM30v yBYrBxDzghOKi3xjeFfyHkyWV84FDDqkknr/b6ykiR/EbFIT0DOQpWtb7YAq2s6LVClR MZYQLZrWEvdamC1xEo49PLrxIHpprtS6fmgtiBsBiwkyYs+wb16MHr5y1MTCU6ifim2F rRVA== X-Gm-Message-State: ABuFfojFI1p4baY0puc1gNnipK9ch4sZf+S8Zc1WTqxYJgne/BT6485q dzXgWb+JP1kVXzifxXDXL2GyrQ== X-Google-Smtp-Source: ACcGV62tHwpl/Kv+7BkNDY9OcsKdDl5mge+tYuXb6oAwjAeDflgO2E9SDg1kavgFstsqbtCCXc/00w== X-Received: by 2002:a0c:896d:: with SMTP id 42-v6mr5725249qvq.117.1537996141228; Wed, 26 Sep 2018 14:09:01 -0700 (PDT) Received: from localhost ([107.15.81.208]) by smtp.gmail.com with ESMTPSA id j5-v6sm65173qtb.34.2018.09.26.14.09.00 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 26 Sep 2018 14:09:00 -0700 (PDT) From: Josef Bacik To: kernel-team@fb.com, linux-kernel@vger.kernel.org, hannes@cmpxchg.org, tj@kernel.org, linux-fsdevel@vger.kernel.org, akpm@linux-foundation.org, riel@redhat.com, linux-mm@kvack.org, linux-btrfs@vger.kernel.org Subject: [PATCH 1/9] mm: infrastructure for page fault page caching Date: Wed, 26 Sep 2018 17:08:48 -0400 Message-Id: <20180926210856.7895-2-josef@toxicpanda.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180926210856.7895-1-josef@toxicpanda.com> References: <20180926210856.7895-1-josef@toxicpanda.com> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP We want to be able to cache the result of a previous loop of a page fault in the case that we use VM_FAULT_RETRY, so introduce handle_mm_fault_cacheable that will take a struct vm_fault directly, add a ->cached_page field to vm_fault, and add helpers to init/cleanup the struct vm_fault. I've converted x86, other arch's can follow suit if they so wish, it's relatively straightforward. Signed-off-by: Josef Bacik --- arch/x86/mm/fault.c | 6 +++- include/linux/mm.h | 31 +++++++++++++++++++++ mm/memory.c | 79 ++++++++++++++++++++++++++++++++--------------------- 3 files changed, 84 insertions(+), 32 deletions(-) diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 47bebfe6efa7..ef6e538c4931 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1211,6 +1211,7 @@ static noinline void __do_page_fault(struct pt_regs *regs, unsigned long error_code, unsigned long address) { + struct vm_fault vmf = {}; struct vm_area_struct *vma; struct task_struct *tsk; struct mm_struct *mm; @@ -1392,7 +1393,8 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code, * fault, so we read the pkey beforehand. */ pkey = vma_pkey(vma); - fault = handle_mm_fault(vma, address, flags); + vm_fault_init(&vmf, vma, address, flags); + fault = handle_mm_fault_cacheable(&vmf); major |= fault & VM_FAULT_MAJOR; /* @@ -1408,6 +1410,7 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code, if (!fatal_signal_pending(tsk)) goto retry; } + vm_fault_cleanup(&vmf); /* User mode? Just return to handle the fatal exception */ if (flags & FAULT_FLAG_USER) @@ -1418,6 +1421,7 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code, return; } + vm_fault_cleanup(&vmf); up_read(&mm->mmap_sem); if (unlikely(fault & VM_FAULT_ERROR)) { mm_fault_error(regs, error_code, address, &pkey, fault); diff --git a/include/linux/mm.h b/include/linux/mm.h index a61ebe8ad4ca..4a84ec976dfc 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -360,6 +360,12 @@ struct vm_fault { * is set (which is also implied by * VM_FAULT_ERROR). */ + struct page *cached_page; /* ->fault handlers that return + * VM_FAULT_RETRY can store their + * previous page here to be reused the + * next time we loop through the fault + * handler for faster lookup. + */ /* These three entries are valid only while holding ptl lock */ pte_t *pte; /* Pointer to pte entry matching * the 'address'. NULL if the page @@ -378,6 +384,16 @@ struct vm_fault { */ }; +static inline void vm_fault_init(struct vm_fault *vmf, + struct vm_area_struct *vma, + unsigned long address, + unsigned int flags) +{ + vmf->vma = vma; + vmf->address = address; + vmf->flags = flags; +} + /* page entry size for vm->huge_fault() */ enum page_entry_size { PE_SIZE_PTE = 0, @@ -943,6 +959,14 @@ static inline void put_page(struct page *page) __put_page(page); } +static inline void vm_fault_cleanup(struct vm_fault *vmf) +{ + if (vmf->cached_page) { + put_page(vmf->cached_page); + vmf->cached_page = NULL; + } +} + #if defined(CONFIG_SPARSEMEM) && !defined(CONFIG_SPARSEMEM_VMEMMAP) #define SECTION_IN_PAGE_FLAGS #endif @@ -1405,6 +1429,7 @@ int invalidate_inode_page(struct page *page); #ifdef CONFIG_MMU extern vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, unsigned int flags); +extern vm_fault_t handle_mm_fault_cacheable(struct vm_fault *vmf); extern int fixup_user_fault(struct task_struct *tsk, struct mm_struct *mm, unsigned long address, unsigned int fault_flags, bool *unlocked); @@ -1420,6 +1445,12 @@ static inline vm_fault_t handle_mm_fault(struct vm_area_struct *vma, BUG(); return VM_FAULT_SIGBUS; } +static inline vm_fault_t handle_mm_fault_cacheable(struct vm_fault *vmf) +{ + /* should never happen if there's no MMU */ + BUG(); + return VM_FAULT_SIGBUS; +} static inline int fixup_user_fault(struct task_struct *tsk, struct mm_struct *mm, unsigned long address, unsigned int fault_flags, bool *unlocked) diff --git a/mm/memory.c b/mm/memory.c index c467102a5cbc..433075f722ea 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4024,36 +4024,34 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) * The mmap_sem may have been released depending on flags and our * return value. See filemap_fault() and __lock_page_or_retry(). */ -static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, - unsigned long address, unsigned int flags) +static vm_fault_t __handle_mm_fault(struct vm_fault *vmf) { - struct vm_fault vmf = { - .vma = vma, - .address = address & PAGE_MASK, - .flags = flags, - .pgoff = linear_page_index(vma, address), - .gfp_mask = __get_fault_gfp_mask(vma), - }; - unsigned int dirty = flags & FAULT_FLAG_WRITE; + struct vm_area_struct *vma = vmf->vma; + unsigned long address = vmf->address; + unsigned int dirty = vmf->flags & FAULT_FLAG_WRITE; struct mm_struct *mm = vma->vm_mm; pgd_t *pgd; p4d_t *p4d; vm_fault_t ret; + vmf->address = address & PAGE_MASK; + vmf->pgoff = linear_page_index(vma, address); + vmf->gfp_mask = __get_fault_gfp_mask(vma); + pgd = pgd_offset(mm, address); p4d = p4d_alloc(mm, pgd, address); if (!p4d) return VM_FAULT_OOM; - vmf.pud = pud_alloc(mm, p4d, address); - if (!vmf.pud) + vmf->pud = pud_alloc(mm, p4d, address); + if (!vmf->pud) return VM_FAULT_OOM; - if (pud_none(*vmf.pud) && transparent_hugepage_enabled(vma)) { - ret = create_huge_pud(&vmf); + if (pud_none(*vmf->pud) && transparent_hugepage_enabled(vma)) { + ret = create_huge_pud(vmf); if (!(ret & VM_FAULT_FALLBACK)) return ret; } else { - pud_t orig_pud = *vmf.pud; + pud_t orig_pud = *vmf->pud; barrier(); if (pud_trans_huge(orig_pud) || pud_devmap(orig_pud)) { @@ -4061,50 +4059,50 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, /* NUMA case for anonymous PUDs would go here */ if (dirty && !pud_write(orig_pud)) { - ret = wp_huge_pud(&vmf, orig_pud); + ret = wp_huge_pud(vmf, orig_pud); if (!(ret & VM_FAULT_FALLBACK)) return ret; } else { - huge_pud_set_accessed(&vmf, orig_pud); + huge_pud_set_accessed(vmf, orig_pud); return 0; } } } - vmf.pmd = pmd_alloc(mm, vmf.pud, address); - if (!vmf.pmd) + vmf->pmd = pmd_alloc(mm, vmf->pud, address); + if (!vmf->pmd) return VM_FAULT_OOM; - if (pmd_none(*vmf.pmd) && transparent_hugepage_enabled(vma)) { - ret = create_huge_pmd(&vmf); + if (pmd_none(*vmf->pmd) && transparent_hugepage_enabled(vma)) { + ret = create_huge_pmd(vmf); if (!(ret & VM_FAULT_FALLBACK)) return ret; } else { - pmd_t orig_pmd = *vmf.pmd; + pmd_t orig_pmd = *vmf->pmd; barrier(); if (unlikely(is_swap_pmd(orig_pmd))) { VM_BUG_ON(thp_migration_supported() && !is_pmd_migration_entry(orig_pmd)); if (is_pmd_migration_entry(orig_pmd)) - pmd_migration_entry_wait(mm, vmf.pmd); + pmd_migration_entry_wait(mm, vmf->pmd); return 0; } if (pmd_trans_huge(orig_pmd) || pmd_devmap(orig_pmd)) { if (pmd_protnone(orig_pmd) && vma_is_accessible(vma)) - return do_huge_pmd_numa_page(&vmf, orig_pmd); + return do_huge_pmd_numa_page(vmf, orig_pmd); if (dirty && !pmd_write(orig_pmd)) { - ret = wp_huge_pmd(&vmf, orig_pmd); + ret = wp_huge_pmd(vmf, orig_pmd); if (!(ret & VM_FAULT_FALLBACK)) return ret; } else { - huge_pmd_set_accessed(&vmf, orig_pmd); + huge_pmd_set_accessed(vmf, orig_pmd); return 0; } } } - return handle_pte_fault(&vmf); + return handle_pte_fault(vmf); } /* @@ -4113,9 +4111,10 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, * The mmap_sem may have been released depending on flags and our * return value. See filemap_fault() and __lock_page_or_retry(). */ -vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, - unsigned int flags) +static vm_fault_t do_handle_mm_fault(struct vm_fault *vmf) { + struct vm_area_struct *vma = vmf->vma; + unsigned int flags = vmf->flags; vm_fault_t ret; __set_current_state(TASK_RUNNING); @@ -4139,9 +4138,9 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, mem_cgroup_enter_user_fault(); if (unlikely(is_vm_hugetlb_page(vma))) - ret = hugetlb_fault(vma->vm_mm, vma, address, flags); + ret = hugetlb_fault(vma->vm_mm, vma, vmf->address, flags); else - ret = __handle_mm_fault(vma, address, flags); + ret = __handle_mm_fault(vmf); if (flags & FAULT_FLAG_USER) { mem_cgroup_exit_user_fault(); @@ -4157,8 +4156,26 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, return ret; } + +vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, + unsigned int flags) +{ + struct vm_fault vmf = {}; + vm_fault_t ret; + + vm_fault_init(&vmf, vma, address, flags); + ret = do_handle_mm_fault(&vmf); + vm_fault_cleanup(&vmf); + return ret; +} EXPORT_SYMBOL_GPL(handle_mm_fault); +vm_fault_t handle_mm_fault_cacheable(struct vm_fault *vmf) +{ + return do_handle_mm_fault(vmf); +} +EXPORT_SYMBOL_GPL(handle_mm_fault_cacheable); + #ifndef __PAGETABLE_P4D_FOLDED /* * Allocate p4d page table. From patchwork Wed Sep 26 21:08:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 10616839 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0C42D174A for ; Wed, 26 Sep 2018 21:09:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F0B7E2B774 for ; Wed, 26 Sep 2018 21:09:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E42C12B7FC; Wed, 26 Sep 2018 21:09:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0D27F2B774 for ; Wed, 26 Sep 2018 21:09:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727171AbeI0DXz (ORCPT ); Wed, 26 Sep 2018 23:23:55 -0400 Received: from mail-qt1-f194.google.com ([209.85.160.194]:39606 "EHLO mail-qt1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727077AbeI0DXy (ORCPT ); Wed, 26 Sep 2018 23:23:54 -0400 Received: by mail-qt1-f194.google.com with SMTP id c25-v6so492998qtp.6 for ; Wed, 26 Sep 2018 14:09:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references; bh=FY4PvfFFgkG4rCGf8NvSi/OKKAo2xpKioHzicQvhcOI=; b=tUlEGY8l8IgAEzc/SrMWSEhdj/YyihWzeRHQm/F7LSmGImuylO2549hO9a14Megk4C S6YLhf0itc2jGfRTXuqZXNNpdLG0n/5fynFIeJx5PCGpk8ve3pr+L3HbN8JwP4oY2vNz are7A4iKZfhnLtonNzI92X7P0gMiAtPr5kJ1P1S0JSLRkB3Xy/YRf40h3yYhGpmrIPrO fINLLRqmeSOAFFhYx/4ZtRJtKCFIbY1d2A4Q0xJO0lqhVB+uTUigu5Wrre1Gp4Gbx/4Q 1/2nAOKAeZ6nAdT/3xXt7evr6JQEeAOsDYjieEmH8aGd86QB9tZKXMuOkAackq2qjcjf pGVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=FY4PvfFFgkG4rCGf8NvSi/OKKAo2xpKioHzicQvhcOI=; b=r2ao7zzDJDJhno+nelDkhPHZhCcpPdTlgZTeY0OM3+4yIvI7FtIWJffx5aHxknPNGX +l7lXbZqfzMt1tuR8+WA0tTRGNmFGFotMVD1wuCAbR6jcr+zZvitTIdAsYYFV3+FPgWS Bm6T2OipdKsLj525HOQvJfIeUrEqJZ5SZ+liKPgwiZ7lK7p4i49b4FGYDOm3TvH3iLgh T7cMZEcGKv6CqmCVikZ5xdeCi6eu0c+bLH1r/x7RCNBDMulhV6+rZCxGYhqZGj1ip4kX dL7Ob70T06nZfYvSMzdlhMVNnXPony90wkoXO42onFWGzE+4TOBnCeHVaCotguu96f+E nnuQ== X-Gm-Message-State: ABuFfohDg5ALxZuVRodkKbA4L+0rEL6rYHiSnEp9NbjCQRublyw9t+Xb HLbqA125iqEpwTSEVtcS9Wv7eQ== X-Google-Smtp-Source: ACcGV63m+uEm9iqYhuq6Vvgdg3700+CAAqavam7D27mFj9y+w+xIfARiMWPqtlMjnrw80RLGv5qyZg== X-Received: by 2002:a0c:f906:: with SMTP id v6-v6mr5821962qvn.86.1537996143023; Wed, 26 Sep 2018 14:09:03 -0700 (PDT) Received: from localhost ([107.15.81.208]) by smtp.gmail.com with ESMTPSA id l18-v6sm96163qtb.2.2018.09.26.14.09.01 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 26 Sep 2018 14:09:02 -0700 (PDT) From: Josef Bacik To: kernel-team@fb.com, linux-kernel@vger.kernel.org, hannes@cmpxchg.org, tj@kernel.org, linux-fsdevel@vger.kernel.org, akpm@linux-foundation.org, riel@redhat.com, linux-mm@kvack.org, linux-btrfs@vger.kernel.org Subject: [PATCH 2/9] mm: drop mmap_sem for page cache read IO submission Date: Wed, 26 Sep 2018 17:08:49 -0400 Message-Id: <20180926210856.7895-3-josef@toxicpanda.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180926210856.7895-1-josef@toxicpanda.com> References: <20180926210856.7895-1-josef@toxicpanda.com> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Johannes Weiner Reads can take a long time, and if anybody needs to take a write lock on the mmap_sem it'll block any subsequent readers to the mmap_sem while the read is outstanding, which could cause long delays. Instead drop the mmap_sem if we do any reads at all. Signed-off-by: Johannes Weiner Signed-off-by: Josef Bacik --- mm/filemap.c | 119 ++++++++++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 90 insertions(+), 29 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 52517f28e6f4..1ed35cd99b2c 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2366,6 +2366,18 @@ generic_file_read_iter(struct kiocb *iocb, struct iov_iter *iter) EXPORT_SYMBOL(generic_file_read_iter); #ifdef CONFIG_MMU +static struct file *maybe_unlock_mmap_for_io(struct vm_area_struct *vma, int flags) +{ + if ((flags & (FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_RETRY_NOWAIT)) == FAULT_FLAG_ALLOW_RETRY) { + struct file *file; + + file = get_file(vma->vm_file); + up_read(&vma->vm_mm->mmap_sem); + return file; + } + return NULL; +} + /** * page_cache_read - adds requested page to the page cache if not already there * @file: file to read @@ -2405,23 +2417,28 @@ static int page_cache_read(struct file *file, pgoff_t offset, gfp_t gfp_mask) * Synchronous readahead happens when we don't even find * a page in the page cache at all. */ -static void do_sync_mmap_readahead(struct vm_area_struct *vma, - struct file_ra_state *ra, - struct file *file, - pgoff_t offset) +static int do_sync_mmap_readahead(struct vm_area_struct *vma, + struct file_ra_state *ra, + struct file *file, + pgoff_t offset, + int flags) { struct address_space *mapping = file->f_mapping; + struct file *fpin; /* If we don't want any read-ahead, don't bother */ if (vma->vm_flags & VM_RAND_READ) - return; + return 0; if (!ra->ra_pages) - return; + return 0; if (vma->vm_flags & VM_SEQ_READ) { + fpin = maybe_unlock_mmap_for_io(vma, flags); page_cache_sync_readahead(mapping, ra, file, offset, ra->ra_pages); - return; + if (fpin) + fput(fpin); + return fpin ? -EAGAIN : 0; } /* Avoid banging the cache line if not needed */ @@ -2433,7 +2450,9 @@ static void do_sync_mmap_readahead(struct vm_area_struct *vma, * stop bothering with read-ahead. It will only hurt. */ if (ra->mmap_miss > MMAP_LOTSAMISS) - return; + return 0; + + fpin = maybe_unlock_mmap_for_io(vma, flags); /* * mmap read-around @@ -2442,28 +2461,40 @@ static void do_sync_mmap_readahead(struct vm_area_struct *vma, ra->size = ra->ra_pages; ra->async_size = ra->ra_pages / 4; ra_submit(ra, mapping, file); + + if (fpin) + fput(fpin); + + return fpin ? -EAGAIN : 0; } /* * Asynchronous readahead happens when we find the page and PG_readahead, * so we want to possibly extend the readahead further.. */ -static void do_async_mmap_readahead(struct vm_area_struct *vma, - struct file_ra_state *ra, - struct file *file, - struct page *page, - pgoff_t offset) +static int do_async_mmap_readahead(struct vm_area_struct *vma, + struct file_ra_state *ra, + struct file *file, + struct page *page, + pgoff_t offset, + int flags) { struct address_space *mapping = file->f_mapping; + struct file *fpin; /* If we don't want any read-ahead, don't bother */ if (vma->vm_flags & VM_RAND_READ) - return; + return 0; if (ra->mmap_miss > 0) ra->mmap_miss--; - if (PageReadahead(page)) - page_cache_async_readahead(mapping, ra, file, - page, offset, ra->ra_pages); + if (!PageReadahead(page)) + return 0; + fpin = maybe_unlock_mmap_for_io(vma, flags); + page_cache_async_readahead(mapping, ra, file, + page, offset, ra->ra_pages); + if (fpin) + fput(fpin); + return fpin ? -EAGAIN : 0; } /** @@ -2479,10 +2510,8 @@ static void do_async_mmap_readahead(struct vm_area_struct *vma, * * vma->vm_mm->mmap_sem must be held on entry. * - * If our return value has VM_FAULT_RETRY set, it's because - * lock_page_or_retry() returned 0. - * The mmap_sem has usually been released in this case. - * See __lock_page_or_retry() for the exception. + * If our return value has VM_FAULT_RETRY set, the mmap_sem has + * usually been released. * * If our return value does not have VM_FAULT_RETRY set, the mmap_sem * has not been released. @@ -2492,11 +2521,13 @@ static void do_async_mmap_readahead(struct vm_area_struct *vma, vm_fault_t filemap_fault(struct vm_fault *vmf) { int error; + struct mm_struct *mm = vmf->vma->vm_mm; struct file *file = vmf->vma->vm_file; struct address_space *mapping = file->f_mapping; struct file_ra_state *ra = &file->f_ra; struct inode *inode = mapping->host; pgoff_t offset = vmf->pgoff; + int flags = vmf->flags; pgoff_t max_off; struct page *page; vm_fault_t ret = 0; @@ -2509,27 +2540,44 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) * Do we have something in the page cache already? */ page = find_get_page(mapping, offset); - if (likely(page) && !(vmf->flags & FAULT_FLAG_TRIED)) { + if (likely(page) && !(flags & FAULT_FLAG_TRIED)) { /* * We found the page, so try async readahead before * waiting for the lock. */ - do_async_mmap_readahead(vmf->vma, ra, file, page, offset); + error = do_async_mmap_readahead(vmf->vma, ra, file, page, offset, vmf->flags); + if (error == -EAGAIN) + goto out_retry_wait; } else if (!page) { /* No page in the page cache at all */ - do_sync_mmap_readahead(vmf->vma, ra, file, offset); - count_vm_event(PGMAJFAULT); - count_memcg_event_mm(vmf->vma->vm_mm, PGMAJFAULT); ret = VM_FAULT_MAJOR; + count_vm_event(PGMAJFAULT); + count_memcg_event_mm(mm, PGMAJFAULT); + error = do_sync_mmap_readahead(vmf->vma, ra, file, offset, vmf->flags); + if (error == -EAGAIN) + goto out_retry_wait; retry_find: page = find_get_page(mapping, offset); if (!page) goto no_cached_page; } - if (!lock_page_or_retry(page, vmf->vma->vm_mm, vmf->flags)) { - put_page(page); - return ret | VM_FAULT_RETRY; + if (!trylock_page(page)) { + if (flags & FAULT_FLAG_ALLOW_RETRY) { + if (flags & FAULT_FLAG_RETRY_NOWAIT) + goto out_retry; + up_read(&mm->mmap_sem); + goto out_retry_wait; + } + if (flags & FAULT_FLAG_KILLABLE) { + int ret = __lock_page_killable(page); + + if (ret) { + up_read(&mm->mmap_sem); + goto out_retry; + } + } else + __lock_page(page); } /* Did it get truncated? */ @@ -2607,6 +2655,19 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) /* Things didn't work out. Return zero to tell the mm layer so. */ shrink_readahead_size_eio(file, ra); return VM_FAULT_SIGBUS; + +out_retry_wait: + if (page) { + if (flags & FAULT_FLAG_KILLABLE) + wait_on_page_locked_killable(page); + else + wait_on_page_locked(page); + } + +out_retry: + if (page) + put_page(page); + return ret | VM_FAULT_RETRY; } EXPORT_SYMBOL(filemap_fault); From patchwork Wed Sep 26 21:08:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 10616883 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8745E3CF1 for ; Wed, 26 Sep 2018 21:10:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7A8D02B7F2 for ; Wed, 26 Sep 2018 21:10:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6EA172B7FC; Wed, 26 Sep 2018 21:10:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C9A212B7F2 for ; Wed, 26 Sep 2018 21:10:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727213AbeI0DX5 (ORCPT ); Wed, 26 Sep 2018 23:23:57 -0400 Received: from mail-qt1-f195.google.com ([209.85.160.195]:34040 "EHLO mail-qt1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727175AbeI0DX4 (ORCPT ); Wed, 26 Sep 2018 23:23:56 -0400 Received: by mail-qt1-f195.google.com with SMTP id x23-v6so531306qtr.1 for ; Wed, 26 Sep 2018 14:09:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ESNx+ZeXEpnsFX3NMmn2fGNCRcRxntWtJSga04sSIlA=; b=XwT1iJ14b0Y8zOY8wRLonfvQWFerAbtD/ma0JeyNfblznWcTzR6PMdo9Gxvn9vQ8Fo uJRpjsCeIPuBeJq5zpQr6RHFH6VBi0clsuwsqTKCsZbmEzxH15vrjtdd8uNMhsbvFzZf Hl8ArPBKxVjv2pIQKeQFmmkEU5eTC6L+jYdtM6fXnAMAHjQiSx39qKqFn+r0NaffcCgc PkTpgRAn0fmPpVCVO3ysC9zDbghlWIBiP7vkQyI7tP5/VAZp0Onh02N/aWIbPJXUVJND pTNKQcRXTFfCsy4cMFQUzmTIBL1Eekk3mAfbRQPKS/SRB2RZE8pGpEwD+vUVG8dtSp+h OvSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ESNx+ZeXEpnsFX3NMmn2fGNCRcRxntWtJSga04sSIlA=; b=R25vjRIvwsXulIWkY5zUowEv315Oiqd+SfUAV8GETcERu/s9giuKOMG8gUYCMDEko/ PGppsX0/tIdjO0OBGS1C1T+HwUOB1go1CIZIqNcXguu0VQHNufWDB00ik/gCZMYsaSOh DjpWjrkra03lN3zaF5LkYU9Ehs6u56g/Vgh6Ur0tavBP9Tg4ive5ZvSAx9WFU3NtcowD K/qjOrKCAhaQucYzS4ZPnwd26ikk4H4L6ePNisdO6HCrxOzGkLNbW9V+I+6CFMxww/FZ v31ufrP2yrwTtJTj0fO/n67Zp3g1McIvjjXyycgVb3V89Il1fcuTA90GAdk8oeWWkv84 1xrw== X-Gm-Message-State: ABuFfogOd6flm8Q11l+9ky+NpOhhj3ByL5tIS2V/F3Gvk+AW3PHh7O7r bwPIrC3Z9Lo0yVxnCmz26VaSFQ== X-Google-Smtp-Source: ACcGV612awagRXq/MVl2xwKC547NGtGmnifm3iz1tSpVjUxR1GQEkPMPqVcQoudSYXdj3D6A8K2fHg== X-Received: by 2002:a0c:9557:: with SMTP id m23-v6mr5789549qvm.138.1537996144815; Wed, 26 Sep 2018 14:09:04 -0700 (PDT) Received: from localhost ([107.15.81.208]) by smtp.gmail.com with ESMTPSA id x11-v6sm63930qtc.48.2018.09.26.14.09.03 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 26 Sep 2018 14:09:03 -0700 (PDT) From: Josef Bacik To: kernel-team@fb.com, linux-kernel@vger.kernel.org, hannes@cmpxchg.org, tj@kernel.org, linux-fsdevel@vger.kernel.org, akpm@linux-foundation.org, riel@redhat.com, linux-mm@kvack.org, linux-btrfs@vger.kernel.org Cc: Johannes Weiner Subject: [PATCH 3/9] mm: clean up swapcache lookup and creation function names Date: Wed, 26 Sep 2018 17:08:50 -0400 Message-Id: <20180926210856.7895-4-josef@toxicpanda.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180926210856.7895-1-josef@toxicpanda.com> References: <20180926210856.7895-1-josef@toxicpanda.com> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Johannes Weiner __read_swap_cache_async() has a misleading name. All it does is look up or create a page in swapcache; it doesn't initiate any IO. The swapcache has many parallels to the page cache, and shares naming schemes with it elsewhere. Analogous to the cache lookup and creation API, rename __read_swap_cache_async() find_or_create_swap_cache() and lookup_swap_cache() to find_swap_cache(). Signed-off-by: Johannes Weiner Signed-off-by: Josef Bacik --- include/linux/swap.h | 14 ++++++++------ mm/memory.c | 2 +- mm/shmem.c | 2 +- mm/swap_state.c | 43 ++++++++++++++++++++++--------------------- mm/zswap.c | 8 ++++---- 5 files changed, 36 insertions(+), 33 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 8e2c11e692ba..293a84c34448 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -412,15 +412,17 @@ extern void __delete_from_swap_cache(struct page *); extern void delete_from_swap_cache(struct page *); extern void free_page_and_swap_cache(struct page *); extern void free_pages_and_swap_cache(struct page **, int); -extern struct page *lookup_swap_cache(swp_entry_t entry, - struct vm_area_struct *vma, - unsigned long addr); +extern struct page *find_swap_cache(swp_entry_t entry, + struct vm_area_struct *vma, + unsigned long addr); +extern struct page *find_or_create_swap_cache(swp_entry_t entry, + gfp_t gfp_mask, + struct vm_area_struct *vma, + unsigned long addr, + bool *created); extern struct page *read_swap_cache_async(swp_entry_t, gfp_t, struct vm_area_struct *vma, unsigned long addr, bool do_poll); -extern struct page *__read_swap_cache_async(swp_entry_t, gfp_t, - struct vm_area_struct *vma, unsigned long addr, - bool *new_page_allocated); extern struct page *swap_cluster_readahead(swp_entry_t entry, gfp_t flag, struct vm_fault *vmf); extern struct page *swapin_readahead(swp_entry_t entry, gfp_t flag, diff --git a/mm/memory.c b/mm/memory.c index 433075f722ea..6f8abde84986 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2935,7 +2935,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) delayacct_set_flag(DELAYACCT_PF_SWAPIN); - page = lookup_swap_cache(entry, vma, vmf->address); + page = find_swap_cache(entry, vma, vmf->address); swapcache = page; if (!page) { diff --git a/mm/shmem.c b/mm/shmem.c index 0376c124b043..9854903ae92f 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1679,7 +1679,7 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, if (swap.val) { /* Look it up and read it in.. */ - page = lookup_swap_cache(swap, NULL, 0); + page = find_swap_cache(swap, NULL, 0); if (!page) { /* Or update major stats only when swapin succeeds?? */ if (fault_type) { diff --git a/mm/swap_state.c b/mm/swap_state.c index ecee9c6c4cc1..bae758e19f7a 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -330,8 +330,8 @@ static inline bool swap_use_vma_readahead(void) * lock getting page table operations atomic even if we drop the page * lock before returning. */ -struct page *lookup_swap_cache(swp_entry_t entry, struct vm_area_struct *vma, - unsigned long addr) +struct page *find_swap_cache(swp_entry_t entry, struct vm_area_struct *vma, + unsigned long addr) { struct page *page; @@ -374,19 +374,20 @@ struct page *lookup_swap_cache(swp_entry_t entry, struct vm_area_struct *vma, return page; } -struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, +struct page *find_or_create_swap_cache(swp_entry_t entry, gfp_t gfp_mask, struct vm_area_struct *vma, unsigned long addr, - bool *new_page_allocated) + bool *created) { struct page *found_page, *new_page = NULL; struct address_space *swapper_space = swap_address_space(entry); int err; - *new_page_allocated = false; + + *created = false; do { /* * First check the swap cache. Since this is normally - * called after lookup_swap_cache() failed, re-calling + * called after find_swap_cache() failed, re-calling * that would confuse statistics. */ found_page = find_get_page(swapper_space, swp_offset(entry)); @@ -449,7 +450,7 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, * Initiate read into locked page and return. */ lru_cache_add_anon(new_page); - *new_page_allocated = true; + *created = true; return new_page; } radix_tree_preload_end(); @@ -475,14 +476,14 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, struct page *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, struct vm_area_struct *vma, unsigned long addr, bool do_poll) { - bool page_was_allocated; - struct page *retpage = __read_swap_cache_async(entry, gfp_mask, - vma, addr, &page_was_allocated); + struct page *page; + bool created; - if (page_was_allocated) - swap_readpage(retpage, do_poll); + page = find_or_create_swap_cache(entry, gfp_mask, vma, addr, &created); + if (created) + swap_readpage(page, do_poll); - return retpage; + return page; } static unsigned int __swapin_nr_pages(unsigned long prev_offset, @@ -573,7 +574,7 @@ struct page *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, unsigned long mask; struct swap_info_struct *si = swp_swap_info(entry); struct blk_plug plug; - bool do_poll = true, page_allocated; + bool do_poll = true, created; struct vm_area_struct *vma = vmf->vma; unsigned long addr = vmf->address; @@ -593,12 +594,12 @@ struct page *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, blk_start_plug(&plug); for (offset = start_offset; offset <= end_offset ; offset++) { /* Ok, do the async read-ahead now */ - page = __read_swap_cache_async( + page = find_or_create_swap_cache( swp_entry(swp_type(entry), offset), - gfp_mask, vma, addr, &page_allocated); + gfp_mask, vma, addr, &created); if (!page) continue; - if (page_allocated) { + if (created) { swap_readpage(page, false); if (offset != entry_offset) { SetPageReadahead(page); @@ -738,7 +739,7 @@ static struct page *swap_vma_readahead(swp_entry_t fentry, gfp_t gfp_mask, pte_t *pte, pentry; swp_entry_t entry; unsigned int i; - bool page_allocated; + bool created; struct vma_swap_readahead ra_info = {0,}; swap_ra_info(vmf, &ra_info); @@ -756,11 +757,11 @@ static struct page *swap_vma_readahead(swp_entry_t fentry, gfp_t gfp_mask, entry = pte_to_swp_entry(pentry); if (unlikely(non_swap_entry(entry))) continue; - page = __read_swap_cache_async(entry, gfp_mask, vma, - vmf->address, &page_allocated); + page = find_or_create_swap_cache(entry, gfp_mask, vma, + vmf->address, &created); if (!page) continue; - if (page_allocated) { + if (created) { swap_readpage(page, false); if (i != ra_info.offset) { SetPageReadahead(page); diff --git a/mm/zswap.c b/mm/zswap.c index cd91fd9d96b8..6f05faa75766 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -823,11 +823,11 @@ enum zswap_get_swap_ret { static int zswap_get_swap_cache_page(swp_entry_t entry, struct page **retpage) { - bool page_was_allocated; + bool created; - *retpage = __read_swap_cache_async(entry, GFP_KERNEL, - NULL, 0, &page_was_allocated); - if (page_was_allocated) + *retpage = find_or_create_swap_cache(entry, GFP_KERNEL, + NULL, 0, &created); + if (created) return ZSWAP_SWAPCACHE_NEW; if (!*retpage) return ZSWAP_SWAPCACHE_FAIL; From patchwork Wed Sep 26 21:08:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 10616845 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D852015A6 for ; Wed, 26 Sep 2018 21:09:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C9BE32B774 for ; Wed, 26 Sep 2018 21:09:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BD5822B7F2; Wed, 26 Sep 2018 21:09:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 628DC2B774 for ; Wed, 26 Sep 2018 21:09:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727266AbeI0DX6 (ORCPT ); Wed, 26 Sep 2018 23:23:58 -0400 Received: from mail-qt1-f196.google.com ([209.85.160.196]:43505 "EHLO mail-qt1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727182AbeI0DX5 (ORCPT ); Wed, 26 Sep 2018 23:23:57 -0400 Received: by mail-qt1-f196.google.com with SMTP id l16-v6so465495qtq.10 for ; Wed, 26 Sep 2018 14:09:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=L4OEsFz3WUjgEGzu73gT/wwAqaCABq/L1QMv8mr7jYM=; b=KzuvBbuaLur8lPktdpvFa9kCRzdghPnke7Nb7qsvyTs/r/mFVoWLTwlp7SbvK6dod8 h27MFYZuD9ST4VbzRwOEKAPER5XjWL9YfkQPGPGFFKJJ6eZmg02p6dUFJwEGkOo0Rp/o 2s+oI3lPekVrwV9EYh0RZoBwGzue0DgWZSj+x5BHKmtXeppLEZBHNBbQx5h4xFeThEDS Wmq33Kk99E24OGVVW9Sd2miTsmwdg2muGlBP6bfTZMxpMudzwt1BTKmf3v+h4Wh1UIML UqAzfJT2d+TRIzWYbwEfQxFoEoEYoAlW1oHwwHLIcspURIeynYfIPaJNnSCLoD45CrDD LZXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=L4OEsFz3WUjgEGzu73gT/wwAqaCABq/L1QMv8mr7jYM=; b=jojXI2M8fuA7ZiKSLsZYeaVgXX5tW4w5/Rz4x4WJUQ/eIivtJH9OAVdWvL7OXHkGGV c0i72nDYdGc3pRIwfTswlVgUYdmNQBKY3QdEThRoAZnJsx0LCach12tb9s4TjKTXAGqh 0Kh2F80oLKKfiLYMfkeHl9Z5ky9BpiP6QXbnVfAw0JCMTxb117qsheSEQfDw+c6OhQpe Mql+C+FiHFOBWXYhdh1UyspnXkOLzw+n32oei0YplKcB6UI/RnXmTKahZVs+GS2S+XUy wtWW56zrw0/5e8uq+25TLx7ttwSZQDwkDgH/sRvVdoHEDEwtXMvf62nWC78nqfOZAdLL Kq+A== X-Gm-Message-State: ABuFfojS8YmHo9Cgl6lXZlsbdO3b3PgA+DGY8mZt16dpjr+ecf5cgF0S bEm74DrXM6clciV1H0RF48HSGw== X-Google-Smtp-Source: ACcGV634tXLL0I1iexAP2WhHj2bvmdrFTXsl3OPvRmlNb/A+m920mCwkcsQS0SkprH9fcqZ0jS0BqQ== X-Received: by 2002:a0c:f906:: with SMTP id v6-v6mr5822169qvn.86.1537996146591; Wed, 26 Sep 2018 14:09:06 -0700 (PDT) Received: from localhost ([107.15.81.208]) by smtp.gmail.com with ESMTPSA id n25-v6sm87327qkl.8.2018.09.26.14.09.05 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 26 Sep 2018 14:09:05 -0700 (PDT) From: Josef Bacik To: kernel-team@fb.com, linux-kernel@vger.kernel.org, hannes@cmpxchg.org, tj@kernel.org, linux-fsdevel@vger.kernel.org, akpm@linux-foundation.org, riel@redhat.com, linux-mm@kvack.org, linux-btrfs@vger.kernel.org Cc: Johannes Weiner Subject: [PATCH 4/9] mm: drop mmap_sem for swap read IO submission Date: Wed, 26 Sep 2018 17:08:51 -0400 Message-Id: <20180926210856.7895-5-josef@toxicpanda.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180926210856.7895-1-josef@toxicpanda.com> References: <20180926210856.7895-1-josef@toxicpanda.com> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Johannes Weiner We don't need to hold the mmap_sem while we're doing the IO, simply drop it and retry appropriately. Signed-off-by: Johannes Weiner Signed-off-by: Josef Bacik --- mm/page_io.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/mm/page_io.c b/mm/page_io.c index aafd19ec1db4..bf21b56a964e 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -365,6 +365,20 @@ int swap_readpage(struct page *page, bool synchronous) goto out; } + /* + * XXX: + * + * Propagate mm->mmap_sem into this function. Then: + * + * get_file(sis->swap_file) + * up_read(mm->mmap_sem) + * submit io request + * fput + * + * After mmap_sem is dropped, sis is no longer valid. Go + * through swap_file->blah->bdev. + */ + if (sis->flags & SWP_FILE) { struct file *swap_file = sis->swap_file; struct address_space *mapping = swap_file->f_mapping; From patchwork Wed Sep 26 21:08:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 10616849 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1E468174A for ; Wed, 26 Sep 2018 21:09:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0F65E2B7F2 for ; Wed, 26 Sep 2018 21:09:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 03CB22B7F8; Wed, 26 Sep 2018 21:09:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E76AD2B774 for ; Wed, 26 Sep 2018 21:09:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727281AbeI0DYB (ORCPT ); Wed, 26 Sep 2018 23:24:01 -0400 Received: from mail-qk1-f196.google.com ([209.85.222.196]:42152 "EHLO mail-qk1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727077AbeI0DX7 (ORCPT ); Wed, 26 Sep 2018 23:23:59 -0400 Received: by mail-qk1-f196.google.com with SMTP id w73-v6so275504qkb.9 for ; Wed, 26 Sep 2018 14:09:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references; bh=fxF9egp/0MfNC6eP3oXuHL949t8miVwsCl5P35THCQ0=; b=xOG6WUhzNHnJDZ8nzE96i5QNtT6Li8Q5zD7JZrUCY2qx9cUIzLHEP5+zgNwBSUtFyp Vsfjd4p80uZmD0X8I4mjGdoXPyZvnafq7f/RBkRbGjEddLbwCS/WwDEsPVkieZMO3WT7 9GYB6qMA/ROB+5KW8+jhZX3ScR0o+/oJHMadB8CL19veYAyrsYcy3UDgvQdXAdmVLpkp 3Lih8/OrYQEWboL0v9vg8SCRCFeaSEwiqoZebAIL5B0MyFM2arHx3endVqpa2rnLVSFF n40P6Ou4BESYXohRryM+Ny3UOyY2G+mJqJ3eWtgKvm8hpnmXLoKe2UZSeAxwErLr5ICk dqww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=fxF9egp/0MfNC6eP3oXuHL949t8miVwsCl5P35THCQ0=; b=Zqbm2jOSDhoiCSoAz8pkPHS6kbkZ2koTk4txSXkNf8CiAKYYfoSM+5JhcX+IVu1Oem rRtJcvnb6Yi16u0uH6JV9JG41944BgHcJSFGrQuXoJqCN5SfvFtKTHsGp0scq3k0iuRP ddjSNvgo8LMlF/OePu9mksQxrq32m+BzmGLs3oq/wkkXNQ6JHo/q7zwVLCi7CTuO4+hH cepxZwgOTYgamEbk7Jn83EJOZZCMaSOAWxmebtFQgkSFBukwruUQ9H7lgjUgGKGrR86v UQfVTPT1A+OfpuuJrLnfrfSR97SFBcF6F72WrVGqaLWthtWlii6aREfE+rZoeKnUIeVD cfOQ== X-Gm-Message-State: ABuFfohn7rFWv9OMmrRSFItfO0wX8ZcJijKr6vkF1Y4p1EvI0dTsBQte P4sshpDNvEvPrtaP50siyBlWYw== X-Google-Smtp-Source: ACcGV62wZZcLGsIgIA0AgxrBDu/rXm6AEl8TqHyyC/dL/kabdkKJb3x4NckjbVpljlp6VQXrWxsz0A== X-Received: by 2002:ae9:e892:: with SMTP id a140-v6mr5494846qkg.242.1537996148385; Wed, 26 Sep 2018 14:09:08 -0700 (PDT) Received: from localhost ([107.15.81.208]) by smtp.gmail.com with ESMTPSA id o7-v6sm50246qkc.67.2018.09.26.14.09.07 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 26 Sep 2018 14:09:07 -0700 (PDT) From: Josef Bacik To: kernel-team@fb.com, linux-kernel@vger.kernel.org, hannes@cmpxchg.org, tj@kernel.org, linux-fsdevel@vger.kernel.org, akpm@linux-foundation.org, riel@redhat.com, linux-mm@kvack.org, linux-btrfs@vger.kernel.org Subject: [PATCH 5/9] mm: drop the mmap_sem in all read fault cases Date: Wed, 26 Sep 2018 17:08:52 -0400 Message-Id: <20180926210856.7895-6-josef@toxicpanda.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180926210856.7895-1-josef@toxicpanda.com> References: <20180926210856.7895-1-josef@toxicpanda.com> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Johannes' patches didn't quite cover all of the IO cases that we need to drop the mmap_sem for, this patch covers the rest of them. Signed-off-by: Josef Bacik --- mm/filemap.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/mm/filemap.c b/mm/filemap.c index 1ed35cd99b2c..65395ee132a0 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2523,6 +2523,7 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) int error; struct mm_struct *mm = vmf->vma->vm_mm; struct file *file = vmf->vma->vm_file; + struct file *fpin = NULL; struct address_space *mapping = file->f_mapping; struct file_ra_state *ra = &file->f_ra; struct inode *inode = mapping->host; @@ -2610,11 +2611,15 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) return ret | VM_FAULT_LOCKED; no_cached_page: + fpin = maybe_unlock_mmap_for_io(vmf->vma, vmf->flags); + /* * We're only likely to ever get here if MADV_RANDOM is in * effect. */ error = page_cache_read(file, offset, vmf->gfp_mask); + if (fpin) + goto out_retry; /* * The page we want has now been added to the page cache. @@ -2634,6 +2639,8 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) return VM_FAULT_SIGBUS; page_not_uptodate: + fpin = maybe_unlock_mmap_for_io(vmf->vma, vmf->flags); + /* * Umm, take care of errors if the page isn't up-to-date. * Try to re-read it _once_. We do this synchronously, @@ -2647,6 +2654,8 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) if (!PageUptodate(page)) error = -EIO; } + if (fpin) + goto out_retry; put_page(page); if (!error || error == AOP_TRUNCATED_PAGE) @@ -2665,6 +2674,8 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) } out_retry: + if (fpin) + fput(fpin); if (page) put_page(page); return ret | VM_FAULT_RETRY; From patchwork Wed Sep 26 21:08:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 10616875 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2C95B14BD for ; Wed, 26 Sep 2018 21:09:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1E8112B7F2 for ; Wed, 26 Sep 2018 21:09:45 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 12FEE2B7FC; Wed, 26 Sep 2018 21:09:45 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B6D8E2B7F2 for ; Wed, 26 Sep 2018 21:09:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727284AbeI0DYB (ORCPT ); Wed, 26 Sep 2018 23:24:01 -0400 Received: from mail-qt1-f174.google.com ([209.85.160.174]:42819 "EHLO mail-qt1-f174.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727272AbeI0DYB (ORCPT ); Wed, 26 Sep 2018 23:24:01 -0400 Received: by mail-qt1-f174.google.com with SMTP id z8-v6so470595qto.9 for ; Wed, 26 Sep 2018 14:09:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references; bh=hGTHjPlv3HOhn5TB2Ipo9k7piqoFCis94sG3VyiCA5k=; b=NSXsHzI/XYShoNL8eGn0laEEoasfKJMhCPMQwHJ2Yvq8bwk04MyKO9oggPZ3BkE18j olp6YBCPQkeZXNp6oFQ9wE8pCqCakgzN1+c7Yf+qmYZnyq8kxdmoMhi8XJ6C2WRQrGQy nRoPCsn5CexUjHhSeDaxBU9NGB0VSELXL9DK0C6Xgh9Ttpe1SYyyAjXUFByg3xAEnsQR o3wyXeLFBnMiBJl6O7+7dZtJdtDKFbvgEAqm2FnuF2w27uQ/bdPH0KQypM/fZS0r/kzB YsM3KwlXFUWsqcyPrPhoIeoekAN5UBtpHcpIjSVtC5HZe9iiYCCrMpVAVC43s00HglYC UBCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=hGTHjPlv3HOhn5TB2Ipo9k7piqoFCis94sG3VyiCA5k=; b=n8mXlIbjdy7oAUBVjZu2Bge4EJzwCrHjmMyUE6r8FgHe9d12ycmWAemRZZI263w2Dp +/A1Kr59l52vpspLwrVuPQMGMgXNqrDdhz2yktIGUVA70V6dsciCCne08AOoDTTXN2sN lHbCicmXr8XLvas2QbNVG43UsD9o8ecHoGTIbg0qQaLMYoMxYrIZzPy2gHpkI5rIx9HB XNbPMsvun1ekE8TDOhEmiXzGXq3YEwqyKYwquLvTpzuiNldjhJbpQJlMWtdSDmYJyNE8 z9AY5rGdDVAyDbp0EOBlOPBbhTGrB18UVIzkHyTAVMT1V7DuVSATcgKSN2sWgIXXQ9ab 24MA== X-Gm-Message-State: ABuFfogTVoPzvkVAGjRPLFDoI6GtKbnUIuPgekcJ5+jJqaIDDM45ObUw J5TnSa/iVhhmbxDc0gh7+Zpntg== X-Google-Smtp-Source: ACcGV63X6GJlDkWL3trIfKsJDYg6xMkxeDmRSldNmGu6BvcZd3OT+4MyGR7tfq+GMXsjHNWbVQwYBQ== X-Received: by 2002:ac8:2672:: with SMTP id v47-v6mr5865520qtv.287.1537996150087; Wed, 26 Sep 2018 14:09:10 -0700 (PDT) Received: from localhost ([107.15.81.208]) by smtp.gmail.com with ESMTPSA id y143-v6sm33940qky.95.2018.09.26.14.09.09 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 26 Sep 2018 14:09:09 -0700 (PDT) From: Josef Bacik To: kernel-team@fb.com, linux-kernel@vger.kernel.org, hannes@cmpxchg.org, tj@kernel.org, linux-fsdevel@vger.kernel.org, akpm@linux-foundation.org, riel@redhat.com, linux-mm@kvack.org, linux-btrfs@vger.kernel.org Subject: [PATCH 6/9] mm: use the cached page for filemap_fault Date: Wed, 26 Sep 2018 17:08:53 -0400 Message-Id: <20180926210856.7895-7-josef@toxicpanda.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180926210856.7895-1-josef@toxicpanda.com> References: <20180926210856.7895-1-josef@toxicpanda.com> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP If we drop the mmap_sem we have to redo the vma lookup which requires redoing the fault handler. Chances are we will just come back to the same page, so save this page in our vmf->cached_page and reuse it in the next loop through the fault handler. Signed-off-by: Josef Bacik --- mm/filemap.c | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 65395ee132a0..5212ab637832 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2530,13 +2530,38 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) pgoff_t offset = vmf->pgoff; int flags = vmf->flags; pgoff_t max_off; - struct page *page; + struct page *page = NULL; + struct page *cached_page = vmf->cached_page; vm_fault_t ret = 0; max_off = DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE); if (unlikely(offset >= max_off)) return VM_FAULT_SIGBUS; + /* + * We may have read in the page already and have a page from an earlier + * loop. If so we need to see if this page is still valid, and if not + * do the whole dance over again. + */ + if (cached_page) { + if (flags & FAULT_FLAG_KILLABLE) { + error = lock_page_killable(cached_page); + if (error) { + up_read(&mm->mmap_sem); + goto out_retry; + } + } else + lock_page(cached_page); + vmf->cached_page = NULL; + if (cached_page->mapping == mapping && + cached_page->index == offset) { + page = cached_page; + goto have_cached_page; + } + unlock_page(cached_page); + put_page(cached_page); + } + /* * Do we have something in the page cache already? */ @@ -2587,6 +2612,7 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) put_page(page); goto retry_find; } +have_cached_page: VM_BUG_ON_PAGE(page->index != offset, page); /* @@ -2677,7 +2703,7 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) if (fpin) fput(fpin); if (page) - put_page(page); + vmf->cached_page = page; return ret | VM_FAULT_RETRY; } EXPORT_SYMBOL(filemap_fault); From patchwork Wed Sep 26 21:08:54 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 10616871 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0F09E180E for ; Wed, 26 Sep 2018 21:09:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 00C542B7F2 for ; Wed, 26 Sep 2018 21:09:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E98E92B7F9; Wed, 26 Sep 2018 21:09:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 98EFE2B7F8 for ; Wed, 26 Sep 2018 21:09:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727326AbeI0DYC (ORCPT ); Wed, 26 Sep 2018 23:24:02 -0400 Received: from mail-qt1-f193.google.com ([209.85.160.193]:38285 "EHLO mail-qt1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727295AbeI0DYC (ORCPT ); Wed, 26 Sep 2018 23:24:02 -0400 Received: by mail-qt1-f193.google.com with SMTP id z13-v6so501652qts.5 for ; Wed, 26 Sep 2018 14:09:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references; bh=rRSaxONrZWxSfcObSyXaZynV5IrpMUJ0BUHUIDN5YaE=; b=cMCnruOrfxbswXpQ+LzDNW4YBVLUWDQs7bq7X5QQbmFe7IBXo+BpnpXZ5nYL+NtYUu UmQOdRfXvT1X04/qw5qRpxkrQHPA9jdosXRIjLUl63CsIpykwMRlwwk/l9e21291y+Il RTMGa1o+lcurVhm+IrAgprHHyRMmR9Q34XHI1p9+jyEySWWfd0/sGQRtQ0sm1r1/OTYR Jn4qDuTHXr1vCpWOEV5xBltDo+2f4ntWFKOsFfmgVmZN08cvjpw0dJcEIgDceEBJqAxt /zQ0Z7X8q4v+RVYpOm8Cnq8jVqw7E/fQDxH+bhVROQZiPF3i35ff81yOhbZzjGw05Nmr vLBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=rRSaxONrZWxSfcObSyXaZynV5IrpMUJ0BUHUIDN5YaE=; b=b4zqRDAha9X18JqGoGuHFNmn5h70Ixs7oEvJ/g6yOGFUp6FeQZdAbVxDvZer2IHTbA a5d1yLIZIZy/Yzm6IlgopmtdsDSaJuxsSR3z6XjT6Ba7Z1hbY5eI3SvJ3BO+f4vjLAO7 pYKVNcgeOhr3VzJPEx0h8GujdmsApmPys5NHjOGLQh7Hm6qljB+EHVRnliRTHaWvdMYq q3IhZBjRu46to+O4hhwE3pdBX3FlUB68I2DYO3OhemaqK8aRT8H0yQlqAfCu8kxhj6rx Vqp2nLIS9I75T5ETsxM2KYvJKEO4kfGQx5MSVOuOzFHOsHfZlF2zbxyrNK+0qZ+3CaKX BHUw== X-Gm-Message-State: ABuFfohyuUWkvGPz6AtOOoZkyURAb3YQjXz4X/CEtXQyBz0xDlgrXw11 HBLipcvucEbhQZDiT73lShRM4ZTjoiY= X-Google-Smtp-Source: ACcGV63nJY6gaLc/Nzii08FEPUVIkCZuTe+OVlkmq03gRhXyS5p+AaMCVfImTqMIHfOrrPvVJ2Pfgg== X-Received: by 2002:ac8:2a81:: with SMTP id b1-v6mr6223154qta.286.1537996151881; Wed, 26 Sep 2018 14:09:11 -0700 (PDT) Received: from localhost ([107.15.81.208]) by smtp.gmail.com with ESMTPSA id u3-v6sm59038qku.46.2018.09.26.14.09.10 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 26 Sep 2018 14:09:10 -0700 (PDT) From: Josef Bacik To: kernel-team@fb.com, linux-kernel@vger.kernel.org, hannes@cmpxchg.org, tj@kernel.org, linux-fsdevel@vger.kernel.org, akpm@linux-foundation.org, riel@redhat.com, linux-mm@kvack.org, linux-btrfs@vger.kernel.org Subject: [PATCH 7/9] mm: add a flag to indicate we used a cached page Date: Wed, 26 Sep 2018 17:08:54 -0400 Message-Id: <20180926210856.7895-8-josef@toxicpanda.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180926210856.7895-1-josef@toxicpanda.com> References: <20180926210856.7895-1-josef@toxicpanda.com> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This is preparation for dropping the mmap_sem in page_mkwrite. We need to know if we used our cached page so we can be sure it is the page we already did the page_mkwrite stuff on so we don't have to redo all of that work. Signed-off-by: Josef Bacik --- include/linux/mm.h | 6 +++++- mm/filemap.c | 5 ++++- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 4a84ec976dfc..a7305d193c71 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -318,6 +318,9 @@ extern pgprot_t protection_map[16]; #define FAULT_FLAG_USER 0x40 /* The fault originated in userspace */ #define FAULT_FLAG_REMOTE 0x80 /* faulting for non current tsk/mm */ #define FAULT_FLAG_INSTRUCTION 0x100 /* The fault was during an instruction fetch */ +#define FAULT_FLAG_USED_CACHED 0x200 /* Our vmf->page was from a previous + * loop through the fault handler. + */ #define FAULT_FLAG_TRACE \ { FAULT_FLAG_WRITE, "WRITE" }, \ @@ -328,7 +331,8 @@ extern pgprot_t protection_map[16]; { FAULT_FLAG_TRIED, "TRIED" }, \ { FAULT_FLAG_USER, "USER" }, \ { FAULT_FLAG_REMOTE, "REMOTE" }, \ - { FAULT_FLAG_INSTRUCTION, "INSTRUCTION" } + { FAULT_FLAG_INSTRUCTION, "INSTRUCTION" }, \ + { FAULT_FLAG_USED_CACHED, "USED_CACHED" } /* * vm_fault is filled by the the pagefault handler and passed to the vma's diff --git a/mm/filemap.c b/mm/filemap.c index 5212ab637832..e9cb44bd35aa 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2556,6 +2556,7 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) if (cached_page->mapping == mapping && cached_page->index == offset) { page = cached_page; + vmf->flags |= FAULT_FLAG_USED_CACHED; goto have_cached_page; } unlock_page(cached_page); @@ -2619,8 +2620,10 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) * We have a locked page in the page cache, now we need to check * that it's up-to-date. If not, it is going to be due to an error. */ - if (unlikely(!PageUptodate(page))) + if (unlikely(!PageUptodate(page))) { + vmf->flags &= ~(FAULT_FLAG_USED_CACHED); goto page_not_uptodate; + } /* * Found the page and have a reference on it. From patchwork Wed Sep 26 21:08:55 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 10616869 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DF019174A for ; Wed, 26 Sep 2018 21:09:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D25632B7F2 for ; Wed, 26 Sep 2018 21:09:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C6E3F2B7FC; Wed, 26 Sep 2018 21:09:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 77FCD2B7F2 for ; Wed, 26 Sep 2018 21:09:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727352AbeI0DYF (ORCPT ); Wed, 26 Sep 2018 23:24:05 -0400 Received: from mail-qt1-f194.google.com ([209.85.160.194]:32972 "EHLO mail-qt1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727296AbeI0DYE (ORCPT ); Wed, 26 Sep 2018 23:24:04 -0400 Received: by mail-qt1-f194.google.com with SMTP id i10-v6so537670qtp.0 for ; Wed, 26 Sep 2018 14:09:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references; bh=uNw1SEfuNh5muyIlKMjO8zXN2XjQ0Df1a1ZQWi9S3hE=; b=E+IkZtxWnldcnfnVAFchZ1hWE504qBGKdX2eijVXxr7G+Yns/Lw/m0dK3zwPRS5BR7 NPhHdsbYsqv7Rk+GL8cmQXek2L9S55HKsJVPA0JzSGx8F5+cBMI0QQ9EHQtfNLwA7i4t sC3pLwKemV+y20kKGmmL7K6hBp6n0IfS+xlPMNJ1jxbDFLwniUzBbzgyaLZ1K5mkWh7P eg9pTRdVcxR0Iz8T8p0VWPa/8okeH8kK+cdMZva7oNTlzpExw4ukGzcJj8CVcUfue3GZ UYlI0UuSGIZmFrnbVaBQ6nzmTHAXCMDv3kggv8aYdCPINTF/+3uSSqUkA/HIfFou9+kA XliA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=uNw1SEfuNh5muyIlKMjO8zXN2XjQ0Df1a1ZQWi9S3hE=; b=af29TQyyzWULFTCAZO8fZ1yMKQLWnOKEDz0f9C5VcmgDimd2N/jx9f2uoCiomZGV/k 0YhtbdjUZBnhR1ahtYXRKjGr5lkiUDviqgWgLQt/+VTTvlTzANm/sBNz3bqZApmehjzV mcv0K5XhorwQBSj2dUoEDn9ZvO8O8JD2835NsvnvAgW22FXgKJ74+TXiT6efzKRJy1Ey rQ/U+LOLXDfXSvkIxm3I8hOaUzTW0Ah01KeWRooSRAdQj/pSDgK0W5B1CH1tiq7vX66o BP/RSip3vEnsIIHgqjNhXbneqpe7c/rgfs+9Tq4cqc9weJXCUtWKDxurIGFp05VRAqiK EN7A== X-Gm-Message-State: ABuFfogngIT3uydGtVV0w49VRYd1N4ITAzbyT33MiNh5fzcW2yvuKC/P 44b+abGiiGk4qgSH2hoq/CsSYA== X-Google-Smtp-Source: ACcGV62nU0p/SpQCuwEhccw6FtSj8cKf4zvclsHPNrLrKJb3VdwuIBbYKgcnnrv2S2qANr8nkWYi+g== X-Received: by 2002:ac8:39a1:: with SMTP id v30-v6mr5494082qte.58.1537996153608; Wed, 26 Sep 2018 14:09:13 -0700 (PDT) Received: from localhost ([107.15.81.208]) by smtp.gmail.com with ESMTPSA id r9-v6sm37064qta.87.2018.09.26.14.09.12 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 26 Sep 2018 14:09:12 -0700 (PDT) From: Josef Bacik To: kernel-team@fb.com, linux-kernel@vger.kernel.org, hannes@cmpxchg.org, tj@kernel.org, linux-fsdevel@vger.kernel.org, akpm@linux-foundation.org, riel@redhat.com, linux-mm@kvack.org, linux-btrfs@vger.kernel.org Subject: [PATCH 8/9] mm: allow ->page_mkwrite to do retries Date: Wed, 26 Sep 2018 17:08:55 -0400 Message-Id: <20180926210856.7895-9-josef@toxicpanda.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180926210856.7895-1-josef@toxicpanda.com> References: <20180926210856.7895-1-josef@toxicpanda.com> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Before we didn't set the retry flag on our vm_fault. We want to allow file systems to drop the mmap_sem if they so choose, so set this flag and deal with VM_FAULT_RETRY appropriately. Signed-off-by: Josef Bacik --- mm/memory.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 6f8abde84986..821435855177 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2384,11 +2384,13 @@ static vm_fault_t do_page_mkwrite(struct vm_fault *vmf) unsigned int old_flags = vmf->flags; vmf->flags = FAULT_FLAG_WRITE|FAULT_FLAG_MKWRITE; + vmf->flags |= old_flags & FAULT_FLAG_ALLOW_RETRY; ret = vmf->vma->vm_ops->page_mkwrite(vmf); /* Restore original flags so that caller is not surprised */ vmf->flags = old_flags; - if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE))) + if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | + VM_FAULT_RETRY))) return ret; if (unlikely(!(ret & VM_FAULT_LOCKED))) { lock_page(page); @@ -2683,7 +2685,8 @@ static vm_fault_t wp_page_shared(struct vm_fault *vmf) pte_unmap_unlock(vmf->pte, vmf->ptl); tmp = do_page_mkwrite(vmf); if (unlikely(!tmp || (tmp & - (VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) { + (VM_FAULT_ERROR | VM_FAULT_NOPAGE | + VM_FAULT_RETRY)))) { put_page(vmf->page); return tmp; } @@ -3716,7 +3719,8 @@ static vm_fault_t do_shared_fault(struct vm_fault *vmf) unlock_page(vmf->page); tmp = do_page_mkwrite(vmf); if (unlikely(!tmp || - (tmp & (VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) { + (tmp & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | + VM_FAULT_RETRY)))) { put_page(vmf->page); return tmp; } From patchwork Wed Sep 26 21:08:56 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 10616859 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 78C03174A for ; Wed, 26 Sep 2018 21:09:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 693E92B7F2 for ; Wed, 26 Sep 2018 21:09:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5D8C52B7F9; Wed, 26 Sep 2018 21:09:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BF3D92B7F2 for ; Wed, 26 Sep 2018 21:09:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727385AbeI0DYH (ORCPT ); Wed, 26 Sep 2018 23:24:07 -0400 Received: from mail-qt1-f196.google.com ([209.85.160.196]:41908 "EHLO mail-qt1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727374AbeI0DYG (ORCPT ); Wed, 26 Sep 2018 23:24:06 -0400 Received: by mail-qt1-f196.google.com with SMTP id m15-v6so478343qtp.8 for ; Wed, 26 Sep 2018 14:09:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references; bh=auspTN6sS8IxHotb0wwOzsGrFw8EJq3KjZuMbDyvHfw=; b=SXAd6Vke7LmvdOzgepTuMkxa5FVG4FDrn4gPRLVbWKiMjOC6JruSWoiGNuqVBPi55i oomIuyIQpOYd5drvE4MP5YkIE/T8CccT7b4a0rCb/jOWcNs/VLzth1hvdpAJIC+Kr8sS Kdql0JzF4IvjxHmTULK+GNHNHU2T1sELYyiUO3fwvg8cB+vzlHY1aCyOkAGeoo7ZuI0K q4tEdtkustrtW0fLoHEhksp0WWm6UWorJUfYwumQBeNgr0t397zLmHZW/2TKCgQPhqn4 LUcY8NcfEvfqRYiCWcGN2kwaHc9TZ9ZIWFZYX+ykxDz+aHy+hpQ5PyNpPQCYmLLRi1PJ GzFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=auspTN6sS8IxHotb0wwOzsGrFw8EJq3KjZuMbDyvHfw=; b=kkA0t9HLkCIUwTowI9DdN6/pt2Q1BVa/gqA25U5LNuDUU00+UNyAL5XqvchdgjMJcY AplrDns/VsVHAQSbwK6JEvk1rAaMb0xb5hTz2ww8UfzbUNKq6Rw78MZrCgWzL8L3KKEd f/dNTGyqqWaCA8ZhQKSwSGdeXGWr8+vifrBVeVHxIY7+IViFgvTpzudjWNLTPdytkfc2 DX3WsevVoay7e1xF5Qe3Yd/KweBMWc3AMtk5Ir2kHTdVRvTeNAkLGTMAJT8ciVD2VL8p NZzYrpJlT7Fv+0diuXzPqpHIjruEndmco0tFG4F2W15gnoCoi8phL3htmqeP+pqtNc4y Mirg== X-Gm-Message-State: ABuFfogpb9DhbyLRgXxUHCHpC3I5W39OUSqRaE0/fDFn2sdD3MtS6UfK e0yMpMyqsiWSgr79wyYfs0807A== X-Google-Smtp-Source: ACcGV62nJC3T0WRvUixrTKzlWB2aS2+o+U13U55Ewpn8iBVNXBNucv90lx3SdSa1/k4WkjXmvdPwbA== X-Received: by 2002:aed:2b43:: with SMTP id p61-v6mr6152561qtd.107.1537996155364; Wed, 26 Sep 2018 14:09:15 -0700 (PDT) Received: from localhost ([107.15.81.208]) by smtp.gmail.com with ESMTPSA id q26-v6sm86927qta.18.2018.09.26.14.09.14 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 26 Sep 2018 14:09:14 -0700 (PDT) From: Josef Bacik To: kernel-team@fb.com, linux-kernel@vger.kernel.org, hannes@cmpxchg.org, tj@kernel.org, linux-fsdevel@vger.kernel.org, akpm@linux-foundation.org, riel@redhat.com, linux-mm@kvack.org, linux-btrfs@vger.kernel.org Subject: [PATCH 9/9] btrfs: drop mmap_sem in mkwrite for btrfs Date: Wed, 26 Sep 2018 17:08:56 -0400 Message-Id: <20180926210856.7895-10-josef@toxicpanda.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180926210856.7895-1-josef@toxicpanda.com> References: <20180926210856.7895-1-josef@toxicpanda.com> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP ->page_mkwrite is extremely expensive in btrfs. We have to reserve space, which can take 6 lifetimes, and we could possibly have to wait on writeback on the page, another several lifetimes. To avoid this simply drop the mmap_sem if we didn't have the cached page and do all of our work and return the appropriate retry error. If we have the cached page we know we did all the right things to set this page up and we can just carry on. Signed-off-by: Josef Bacik --- fs/btrfs/inode.c | 41 +++++++++++++++++++++++++++++++++++++++-- include/linux/mm.h | 14 ++++++++++++++ mm/filemap.c | 3 ++- 3 files changed, 55 insertions(+), 3 deletions(-) diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index 3ea5339603cf..6b723d29bc0c 100644 --- a/fs/btrfs/inode.c +++ b/fs/btrfs/inode.c @@ -8809,7 +8809,9 @@ static void btrfs_invalidatepage(struct page *page, unsigned int offset, vm_fault_t btrfs_page_mkwrite(struct vm_fault *vmf) { struct page *page = vmf->page; - struct inode *inode = file_inode(vmf->vma->vm_file); + struct file *file = vmf->vma->vm_file, *fpin; + struct mm_struct *mm = vmf->vma->vm_mm; + struct inode *inode = file_inode(file); struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb); struct extent_io_tree *io_tree = &BTRFS_I(inode)->io_tree; struct btrfs_ordered_extent *ordered; @@ -8828,6 +8830,29 @@ vm_fault_t btrfs_page_mkwrite(struct vm_fault *vmf) reserved_space = PAGE_SIZE; + /* + * We have our cached page from a previous mkwrite, check it to make + * sure it's still dirty and our file size matches when we ran mkwrite + * the last time. If everything is OK then return VM_FAULT_LOCKED, + * otherwise do the mkwrite again. + */ + if (vmf->flags & FAULT_FLAG_USED_CACHED) { + lock_page(page); + if (vmf->cached_size == i_size_read(inode) && + PageDirty(page)) + return VM_FAULT_LOCKED; + unlock_page(page); + } + + /* + * mkwrite is extremely expensive, and we are holding the mmap_sem + * during this, which means we can starve out anybody trying to + * down_write(mmap_sem) for a long while, especially if we throw cgroups + * into the mix. So just drop the mmap_sem and do all of our work, + * we'll loop back through and verify everything is ok the next time and + * hopefully avoid doing the work twice. + */ + fpin = maybe_unlock_mmap_for_io(vmf->vma, vmf->flags); sb_start_pagefault(inode->i_sb); page_start = page_offset(page); page_end = page_start + PAGE_SIZE - 1; @@ -8844,7 +8869,7 @@ vm_fault_t btrfs_page_mkwrite(struct vm_fault *vmf) ret2 = btrfs_delalloc_reserve_space(inode, &data_reserved, page_start, reserved_space); if (!ret2) { - ret2 = file_update_time(vmf->vma->vm_file); + ret2 = file_update_time(file); reserved = 1; } if (ret2) { @@ -8943,6 +8968,14 @@ vm_fault_t btrfs_page_mkwrite(struct vm_fault *vmf) btrfs_delalloc_release_extents(BTRFS_I(inode), PAGE_SIZE, true); sb_end_pagefault(inode->i_sb); extent_changeset_free(data_reserved); + if (fpin) { + unlock_page(page); + fput(fpin); + get_page(page); + vmf->cached_size = size; + vmf->cached_page = page; + return VM_FAULT_RETRY; + } return VM_FAULT_LOCKED; } @@ -8955,6 +8988,10 @@ vm_fault_t btrfs_page_mkwrite(struct vm_fault *vmf) out_noreserve: sb_end_pagefault(inode->i_sb); extent_changeset_free(data_reserved); + if (fpin) { + fput(fpin); + down_read(&mm->mmap_sem); + } return ret; } diff --git a/include/linux/mm.h b/include/linux/mm.h index a7305d193c71..9409845d0411 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -370,6 +370,13 @@ struct vm_fault { * next time we loop through the fault * handler for faster lookup. */ + loff_t cached_size; /* ->page_mkwrite handlers may drop + * the mmap_sem to avoid starvation, in + * which case they need to save the + * i_size in order to verify the cached + * page we're using the next loop + * through hasn't changed under us. + */ /* These three entries are valid only while holding ptl lock */ pte_t *pte; /* Pointer to pte entry matching * the 'address'. NULL if the page @@ -1437,6 +1444,8 @@ extern vm_fault_t handle_mm_fault_cacheable(struct vm_fault *vmf); extern int fixup_user_fault(struct task_struct *tsk, struct mm_struct *mm, unsigned long address, unsigned int fault_flags, bool *unlocked); +extern struct file *maybe_unlock_mmap_for_io(struct vm_area_struct *vma, + int flags); void unmap_mapping_pages(struct address_space *mapping, pgoff_t start, pgoff_t nr, bool even_cows); void unmap_mapping_range(struct address_space *mapping, @@ -1463,6 +1472,11 @@ static inline int fixup_user_fault(struct task_struct *tsk, BUG(); return -EFAULT; } +stiatc inline struct file *maybe_unlock_mmap_for_io(struct vm_area_struct *vma, + int flags) +{ + return NULL; +} static inline void unmap_mapping_pages(struct address_space *mapping, pgoff_t start, pgoff_t nr, bool even_cows) { } static inline void unmap_mapping_range(struct address_space *mapping, diff --git a/mm/filemap.c b/mm/filemap.c index e9cb44bd35aa..8027f082d74f 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2366,7 +2366,7 @@ generic_file_read_iter(struct kiocb *iocb, struct iov_iter *iter) EXPORT_SYMBOL(generic_file_read_iter); #ifdef CONFIG_MMU -static struct file *maybe_unlock_mmap_for_io(struct vm_area_struct *vma, int flags) +struct file *maybe_unlock_mmap_for_io(struct vm_area_struct *vma, int flags) { if ((flags & (FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_RETRY_NOWAIT)) == FAULT_FLAG_ALLOW_RETRY) { struct file *file; @@ -2377,6 +2377,7 @@ static struct file *maybe_unlock_mmap_for_io(struct vm_area_struct *vma, int fla } return NULL; } +EXPORT_SYMBOL_GPL(maybe_unlock_mmap_for_io); /** * page_cache_read - adds requested page to the page cache if not already there