From patchwork Wed Sep 26 21:08:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 10616837 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ADAAE15A6 for ; Wed, 26 Sep 2018 21:09:06 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9E90D2B774 for ; Wed, 26 Sep 2018 21:09:06 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 927982B7F2; Wed, 26 Sep 2018 21:09:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 666EE2B791 for ; Wed, 26 Sep 2018 21:09:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3E0898E0003; Wed, 26 Sep 2018 17:09:03 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 390F28E0001; Wed, 26 Sep 2018 17:09:03 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1E33A8E0003; Wed, 26 Sep 2018 17:09:03 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by kanga.kvack.org (Postfix) with ESMTP id E7A1D8E0001 for ; Wed, 26 Sep 2018 17:09:02 -0400 (EDT) Received: by mail-qk1-f200.google.com with SMTP id w126-v6so443988qka.11 for ; Wed, 26 Sep 2018 14:09:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:subject:date:message-id :in-reply-to:references; bh=epHd1o6KPupVYmL3VXxvF5Z1JP8sYlPDk5pYju44IUA=; b=GHaMy5iUyL2cFd4A8F1k1KXVB8H5XXwjPgnYptIdwSrbnlpFuqHILPh9wAv0SB+g5v qY0dCedEBPuupdsqHP3cD9s8ke7G63biPrEU+vbKLuUZwFURoJlN5tml6rXp+GqThKw/ 0LzhNdtD2RPiXbfdJVTRzodULZuISGpSR/gXg8BpoCcrG4kAosDBQRQtKhPfaOJ9cWVx eiUI4acjTJOyXj0RmnGD0vlgtG1dxUrPrcIvV1mIdPT80p3p0xZufbA/zWf5UMk8gmOD O454iYfBY2VecrJspZMcUgWtB0UoYlnFQVQO1dEKEvFp99GzoXQ+naEdUs0q9fBsRwHg 77Lw== X-Gm-Message-State: ABuFfohk2LXyW1lea5MeFEhd3Fq53v5ylSGoWoYNTKityp6LLiod+lFv vR8yunnfVvnCG6XecnD+GEYDmfw9oZunW5V8DLQ7pZtCvEAtXZnUiLxC89N9dQy6iwsyQOQscq0 jCsezVYSYi9DFB+AxnNdeg7YyxWmRiu6ODWar4tC/m+ABnY61z3bkt9D/TT0G0pyB41bSNkQCIM De1xQ60VVHfkyPEHptJcAsu/tWuU/cxxM14Fc9vncuWQL/9WywtPnMmsyYctK7is8mgiBybow5f BCLbJFbi7Wj3IqG3qvEpWCap8Zr+PD7qBB5YJOglxI+P0/wpY3vIgoVJiPWzsIDwV6jn4j5SKRX qX3PxDpeJjZkvLib7/VBFCBIJhy+fTWlodPQohTA4GPuC0HOrfig+PDIQEfmjtDQdgylF2EsyeD h X-Received: by 2002:a0c:988a:: with SMTP id f10-v6mr5838798qvd.101.1537996142698; Wed, 26 Sep 2018 14:09:02 -0700 (PDT) X-Received: by 2002:a0c:988a:: with SMTP id f10-v6mr5838742qvd.101.1537996141727; Wed, 26 Sep 2018 14:09:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537996141; cv=none; d=google.com; s=arc-20160816; b=y6NcSJfX6unCbMljaoZ+M+yWKacqfUu5bFm0CvlXy5z1iSpLb1cE1CAKZggRx+cGSg S8rWV/xfZLzNMqQkDyv7p8keNA3VadmNTWbA6nbZAZrB2d2LjvQIt6Q30zobBeKErNMH hcIRvrCGIoYfZ321sVcTHH1wUPw7op5lCY3BD5sbyFiV3RS43cL55ZhbrFkgXrmLIRBZ P9DPY4o0PAK1c474ZFJimISkKI+yqcCQfB6DX2DcvyiypQcyamJOUQucxUjaJgyOGGmV n9Z5KRINOFqKEOLkN2JJNS9CJoUh5SWp9fhEFdXTqhZWdu55cwRajTUlmfrAemeCc9Jn 5zwQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=epHd1o6KPupVYmL3VXxvF5Z1JP8sYlPDk5pYju44IUA=; b=ThwGIO1SUqvvs+Fvxlgcul8AazKLTIIiQ2zPBf+t7lUaw28fOEm4+ZJdoFYtfZqbZl OIagfQ0hq5Syp94qrN9GRlCRC9YbbBIxSLHDRVDTfeEjoZ+mm+Gbj94+dYeIvTBNAIbG Fpl1/ns/wwuEjgM9iVBAmY39RoJ0POkqOFp+ql5Q4EvrnndfekJwZKVgV5iiIPtcMEjW G9W+P0nLJl0jLDQo0FmdumnmW+HKnBFXMNi5iwdnl/n3EGqSJSVT50kB2wNA+J5dkHaS igaNS8BWmLD0F8M+EiINAawcQW4ciiXN0AlvMllAhPTOOBb3e/qXe5/wnH1ASlSnLfQt 9lXw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@toxicpanda-com.20150623.gappssmtp.com header.s=20150623 header.b="C/WmY+fO"; spf=neutral (google.com: 209.85.220.65 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) smtp.mailfrom=josef@toxicpanda.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id q32-v6sor58604qvh.28.2018.09.26.14.09.01 for (Google Transport Security); Wed, 26 Sep 2018 14:09:01 -0700 (PDT) Received-SPF: neutral (google.com: 209.85.220.65 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@toxicpanda-com.20150623.gappssmtp.com header.s=20150623 header.b="C/WmY+fO"; spf=neutral (google.com: 209.85.220.65 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) smtp.mailfrom=josef@toxicpanda.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references; bh=epHd1o6KPupVYmL3VXxvF5Z1JP8sYlPDk5pYju44IUA=; b=C/WmY+fOpp8biQhBYCH31rM3jxFDT8U3+B6k3+P9k8a8uUBjjm3DsNlaVszOpkOIQo MhoaXlP5edZvLpW2AIIau1epngcB3LJW6tcdkvtv62DhvqCZYUDfiMhQ3ztyWn6xckA0 f3tSc1ecKkFzgWbjR2rjl/LEhRkA3YO6uO6jkD8l4Kd72aS1NsJu2cVUB854i7pFgAEQ HkfYEKZvYHwb2g4McN7JDAzlCZ5Khe/LC4AKkNCgcsoVBZHN0iR13srqaEyZdObdP2W1 GWHBvX00aDWXr2TmSdd1W8h4iWKF8Lx93fIMGJksmT2EbeBONfqSXG8ZFqC/OZk/utfH bV2A== X-Google-Smtp-Source: ACcGV62tHwpl/Kv+7BkNDY9OcsKdDl5mge+tYuXb6oAwjAeDflgO2E9SDg1kavgFstsqbtCCXc/00w== X-Received: by 2002:a0c:896d:: with SMTP id 42-v6mr5725249qvq.117.1537996141228; Wed, 26 Sep 2018 14:09:01 -0700 (PDT) Received: from localhost ([107.15.81.208]) by smtp.gmail.com with ESMTPSA id j5-v6sm65173qtb.34.2018.09.26.14.09.00 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 26 Sep 2018 14:09:00 -0700 (PDT) From: Josef Bacik To: kernel-team@fb.com, linux-kernel@vger.kernel.org, hannes@cmpxchg.org, tj@kernel.org, linux-fsdevel@vger.kernel.org, akpm@linux-foundation.org, riel@redhat.com, linux-mm@kvack.org, linux-btrfs@vger.kernel.org Subject: [PATCH 1/9] mm: infrastructure for page fault page caching Date: Wed, 26 Sep 2018 17:08:48 -0400 Message-Id: <20180926210856.7895-2-josef@toxicpanda.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180926210856.7895-1-josef@toxicpanda.com> References: <20180926210856.7895-1-josef@toxicpanda.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP We want to be able to cache the result of a previous loop of a page fault in the case that we use VM_FAULT_RETRY, so introduce handle_mm_fault_cacheable that will take a struct vm_fault directly, add a ->cached_page field to vm_fault, and add helpers to init/cleanup the struct vm_fault. I've converted x86, other arch's can follow suit if they so wish, it's relatively straightforward. Signed-off-by: Josef Bacik --- arch/x86/mm/fault.c | 6 +++- include/linux/mm.h | 31 +++++++++++++++++++++ mm/memory.c | 79 ++++++++++++++++++++++++++++++++--------------------- 3 files changed, 84 insertions(+), 32 deletions(-) diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 47bebfe6efa7..ef6e538c4931 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1211,6 +1211,7 @@ static noinline void __do_page_fault(struct pt_regs *regs, unsigned long error_code, unsigned long address) { + struct vm_fault vmf = {}; struct vm_area_struct *vma; struct task_struct *tsk; struct mm_struct *mm; @@ -1392,7 +1393,8 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code, * fault, so we read the pkey beforehand. */ pkey = vma_pkey(vma); - fault = handle_mm_fault(vma, address, flags); + vm_fault_init(&vmf, vma, address, flags); + fault = handle_mm_fault_cacheable(&vmf); major |= fault & VM_FAULT_MAJOR; /* @@ -1408,6 +1410,7 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code, if (!fatal_signal_pending(tsk)) goto retry; } + vm_fault_cleanup(&vmf); /* User mode? Just return to handle the fatal exception */ if (flags & FAULT_FLAG_USER) @@ -1418,6 +1421,7 @@ __do_page_fault(struct pt_regs *regs, unsigned long error_code, return; } + vm_fault_cleanup(&vmf); up_read(&mm->mmap_sem); if (unlikely(fault & VM_FAULT_ERROR)) { mm_fault_error(regs, error_code, address, &pkey, fault); diff --git a/include/linux/mm.h b/include/linux/mm.h index a61ebe8ad4ca..4a84ec976dfc 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -360,6 +360,12 @@ struct vm_fault { * is set (which is also implied by * VM_FAULT_ERROR). */ + struct page *cached_page; /* ->fault handlers that return + * VM_FAULT_RETRY can store their + * previous page here to be reused the + * next time we loop through the fault + * handler for faster lookup. + */ /* These three entries are valid only while holding ptl lock */ pte_t *pte; /* Pointer to pte entry matching * the 'address'. NULL if the page @@ -378,6 +384,16 @@ struct vm_fault { */ }; +static inline void vm_fault_init(struct vm_fault *vmf, + struct vm_area_struct *vma, + unsigned long address, + unsigned int flags) +{ + vmf->vma = vma; + vmf->address = address; + vmf->flags = flags; +} + /* page entry size for vm->huge_fault() */ enum page_entry_size { PE_SIZE_PTE = 0, @@ -943,6 +959,14 @@ static inline void put_page(struct page *page) __put_page(page); } +static inline void vm_fault_cleanup(struct vm_fault *vmf) +{ + if (vmf->cached_page) { + put_page(vmf->cached_page); + vmf->cached_page = NULL; + } +} + #if defined(CONFIG_SPARSEMEM) && !defined(CONFIG_SPARSEMEM_VMEMMAP) #define SECTION_IN_PAGE_FLAGS #endif @@ -1405,6 +1429,7 @@ int invalidate_inode_page(struct page *page); #ifdef CONFIG_MMU extern vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, unsigned int flags); +extern vm_fault_t handle_mm_fault_cacheable(struct vm_fault *vmf); extern int fixup_user_fault(struct task_struct *tsk, struct mm_struct *mm, unsigned long address, unsigned int fault_flags, bool *unlocked); @@ -1420,6 +1445,12 @@ static inline vm_fault_t handle_mm_fault(struct vm_area_struct *vma, BUG(); return VM_FAULT_SIGBUS; } +static inline vm_fault_t handle_mm_fault_cacheable(struct vm_fault *vmf) +{ + /* should never happen if there's no MMU */ + BUG(); + return VM_FAULT_SIGBUS; +} static inline int fixup_user_fault(struct task_struct *tsk, struct mm_struct *mm, unsigned long address, unsigned int fault_flags, bool *unlocked) diff --git a/mm/memory.c b/mm/memory.c index c467102a5cbc..433075f722ea 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4024,36 +4024,34 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) * The mmap_sem may have been released depending on flags and our * return value. See filemap_fault() and __lock_page_or_retry(). */ -static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, - unsigned long address, unsigned int flags) +static vm_fault_t __handle_mm_fault(struct vm_fault *vmf) { - struct vm_fault vmf = { - .vma = vma, - .address = address & PAGE_MASK, - .flags = flags, - .pgoff = linear_page_index(vma, address), - .gfp_mask = __get_fault_gfp_mask(vma), - }; - unsigned int dirty = flags & FAULT_FLAG_WRITE; + struct vm_area_struct *vma = vmf->vma; + unsigned long address = vmf->address; + unsigned int dirty = vmf->flags & FAULT_FLAG_WRITE; struct mm_struct *mm = vma->vm_mm; pgd_t *pgd; p4d_t *p4d; vm_fault_t ret; + vmf->address = address & PAGE_MASK; + vmf->pgoff = linear_page_index(vma, address); + vmf->gfp_mask = __get_fault_gfp_mask(vma); + pgd = pgd_offset(mm, address); p4d = p4d_alloc(mm, pgd, address); if (!p4d) return VM_FAULT_OOM; - vmf.pud = pud_alloc(mm, p4d, address); - if (!vmf.pud) + vmf->pud = pud_alloc(mm, p4d, address); + if (!vmf->pud) return VM_FAULT_OOM; - if (pud_none(*vmf.pud) && transparent_hugepage_enabled(vma)) { - ret = create_huge_pud(&vmf); + if (pud_none(*vmf->pud) && transparent_hugepage_enabled(vma)) { + ret = create_huge_pud(vmf); if (!(ret & VM_FAULT_FALLBACK)) return ret; } else { - pud_t orig_pud = *vmf.pud; + pud_t orig_pud = *vmf->pud; barrier(); if (pud_trans_huge(orig_pud) || pud_devmap(orig_pud)) { @@ -4061,50 +4059,50 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, /* NUMA case for anonymous PUDs would go here */ if (dirty && !pud_write(orig_pud)) { - ret = wp_huge_pud(&vmf, orig_pud); + ret = wp_huge_pud(vmf, orig_pud); if (!(ret & VM_FAULT_FALLBACK)) return ret; } else { - huge_pud_set_accessed(&vmf, orig_pud); + huge_pud_set_accessed(vmf, orig_pud); return 0; } } } - vmf.pmd = pmd_alloc(mm, vmf.pud, address); - if (!vmf.pmd) + vmf->pmd = pmd_alloc(mm, vmf->pud, address); + if (!vmf->pmd) return VM_FAULT_OOM; - if (pmd_none(*vmf.pmd) && transparent_hugepage_enabled(vma)) { - ret = create_huge_pmd(&vmf); + if (pmd_none(*vmf->pmd) && transparent_hugepage_enabled(vma)) { + ret = create_huge_pmd(vmf); if (!(ret & VM_FAULT_FALLBACK)) return ret; } else { - pmd_t orig_pmd = *vmf.pmd; + pmd_t orig_pmd = *vmf->pmd; barrier(); if (unlikely(is_swap_pmd(orig_pmd))) { VM_BUG_ON(thp_migration_supported() && !is_pmd_migration_entry(orig_pmd)); if (is_pmd_migration_entry(orig_pmd)) - pmd_migration_entry_wait(mm, vmf.pmd); + pmd_migration_entry_wait(mm, vmf->pmd); return 0; } if (pmd_trans_huge(orig_pmd) || pmd_devmap(orig_pmd)) { if (pmd_protnone(orig_pmd) && vma_is_accessible(vma)) - return do_huge_pmd_numa_page(&vmf, orig_pmd); + return do_huge_pmd_numa_page(vmf, orig_pmd); if (dirty && !pmd_write(orig_pmd)) { - ret = wp_huge_pmd(&vmf, orig_pmd); + ret = wp_huge_pmd(vmf, orig_pmd); if (!(ret & VM_FAULT_FALLBACK)) return ret; } else { - huge_pmd_set_accessed(&vmf, orig_pmd); + huge_pmd_set_accessed(vmf, orig_pmd); return 0; } } } - return handle_pte_fault(&vmf); + return handle_pte_fault(vmf); } /* @@ -4113,9 +4111,10 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, * The mmap_sem may have been released depending on flags and our * return value. See filemap_fault() and __lock_page_or_retry(). */ -vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, - unsigned int flags) +static vm_fault_t do_handle_mm_fault(struct vm_fault *vmf) { + struct vm_area_struct *vma = vmf->vma; + unsigned int flags = vmf->flags; vm_fault_t ret; __set_current_state(TASK_RUNNING); @@ -4139,9 +4138,9 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, mem_cgroup_enter_user_fault(); if (unlikely(is_vm_hugetlb_page(vma))) - ret = hugetlb_fault(vma->vm_mm, vma, address, flags); + ret = hugetlb_fault(vma->vm_mm, vma, vmf->address, flags); else - ret = __handle_mm_fault(vma, address, flags); + ret = __handle_mm_fault(vmf); if (flags & FAULT_FLAG_USER) { mem_cgroup_exit_user_fault(); @@ -4157,8 +4156,26 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, return ret; } + +vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, + unsigned int flags) +{ + struct vm_fault vmf = {}; + vm_fault_t ret; + + vm_fault_init(&vmf, vma, address, flags); + ret = do_handle_mm_fault(&vmf); + vm_fault_cleanup(&vmf); + return ret; +} EXPORT_SYMBOL_GPL(handle_mm_fault); +vm_fault_t handle_mm_fault_cacheable(struct vm_fault *vmf) +{ + return do_handle_mm_fault(vmf); +} +EXPORT_SYMBOL_GPL(handle_mm_fault_cacheable); + #ifndef __PAGETABLE_P4D_FOLDED /* * Allocate p4d page table. From patchwork Wed Sep 26 21:08:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 10616841 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9A9DC180E for ; Wed, 26 Sep 2018 21:09:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8CDBD2B774 for ; Wed, 26 Sep 2018 21:09:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 80D982B7F2; Wed, 26 Sep 2018 21:09:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DAFC72B7F9 for ; Wed, 26 Sep 2018 21:09:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B62038E0004; Wed, 26 Sep 2018 17:09:04 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id ABE918E0001; Wed, 26 Sep 2018 17:09:04 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 89B428E0005; Wed, 26 Sep 2018 17:09:04 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by kanga.kvack.org (Postfix) with ESMTP id 5CDA58E0004 for ; Wed, 26 Sep 2018 17:09:04 -0400 (EDT) Received: by mail-qk1-f197.google.com with SMTP id x144-v6so473024qkb.4 for ; Wed, 26 Sep 2018 14:09:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:subject:date:message-id :in-reply-to:references; bh=FY4PvfFFgkG4rCGf8NvSi/OKKAo2xpKioHzicQvhcOI=; b=CRE2yw7ALWQt4ZxXJMDGz/NIuyoFfp9AjDvrQz9r0i5nUCe8nhJsyrQgajw1j0AVwG jOlTFKVdeGtlC6WEUkKVFVSSmz4ycphZQ/Y2yjaqWF1U+rpqVhGuQ+3MsOQ/OAeLRVKv igpYDKpMvBmUuun/KuO95sMg6tx6EjTa1S2iAK1iOv5cJal5xJ6FREhtwVnYWjwXwBkb MBftLumiYd+TMXqYicqHVY9mCLzG4i94SxSdNyureE5cjBMgxBm+7U5nbsE82ZKwvSs0 hJZ+SNB+k0XVCpsk9dLN5P4Ce29zQJkdhjgkIXFwA5a4SxokenBEWxALaJGUKet6c7qj eU3g== X-Gm-Message-State: ABuFfojiPSFRy6KO/4FC/+n69dhl+FOJPOiyjtdORw3fGHq0Kz66YYDl oqfwsXUzWpRnhAk/pXjymn+XnldvzbTLVlwpGc90/7IIsVo3uH2SpVbrBz2ZeP1Yg5hQvKDMypJ 3lRTiXG5STcpM4sl8yxCJ4nzieuF3Q1dSZej+7hR9nkonA6/O0jHEzkni0j5RrgjDI5UexnqggR /x2BIrUfBGXxIuA7Wh/O4HzH3nDtY3Kyi0bSw+8JPD21DiQn3cTMPyu4BaUwEU5KUp0pdvLoy+G cs4G0iCjgJijjX+WU4CsY8dvvpkrpkjZPRem+gy+B2YiLuK3jFZqGfTTMSd/jVAFs2v0Cyge2YQ 76+HFWp04J5lQsfSXrSIvbckFjgh6xZYbgL0wFPWy9XU0rNQqNRhoeKONzUskNaSQiXF6WOaIrf w X-Received: by 2002:ac8:6c8:: with SMTP id j8-v6mr5947980qth.314.1537996144114; Wed, 26 Sep 2018 14:09:04 -0700 (PDT) X-Received: by 2002:ac8:6c8:: with SMTP id j8-v6mr5947926qth.314.1537996143321; Wed, 26 Sep 2018 14:09:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537996143; cv=none; d=google.com; s=arc-20160816; b=s0AUDJDkoYpWUd+EOHp1AlPFDG9iW7HCw+iYkvjChATf94Eyr9g2z5fII6YKY5XwHJ jkiTyOTjZ9oW8brblQjGej9ZHFCyIUHqe6ViL8mKyroem7L0/MAE2lJx+xGh83ah7JKi pLKMQun4kxPHMsiLpyTMDnTjiV7fWDqkLz9zCR036pfqBal213ZZWhhmtHmFhJieE1cB iAcE8lz1q54acgaI0UcPOHaYQIGw07wXhYNTaegQSGYYbcUCt2xUUh87XhLCPZawLZvC VQmxwKVi9UwWmQiTHgyLlkhWbJoyu82md3BihzeKlQbkD35mL2zn7EprTAx38Wf0iase EqoQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=FY4PvfFFgkG4rCGf8NvSi/OKKAo2xpKioHzicQvhcOI=; b=ICb9JqiQDhsJNIXGQYPVzlwBqBwSyMntCs1hPbX2x21w5vvhhjCswwwDye6hkMepo0 1Vhd+DvDucFUhOwruKAP3Nkt+MtkxFhFgw+0hqtITfWwJeDOt/vsUrSXyeT54Mv97K52 gNBoCjvxUisULTmV1CguWBFGHIxob6qp8Bp6wKLbVWaS7B9u+qWzUeEB3R7OlN8sg7WS VwTbokSbCC8PQXu2+h415b2UtKTrcpWUwIo5vMuRTQn1qPxQoIWHc5BsRpdmc4vKnUjl gpEaJ1FrfYnGngEHEfhapUs7yQi7/jqk35aZwOx8NxU1sMf6gh41UxK7MGYBcbNltCaK QggA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@toxicpanda-com.20150623.gappssmtp.com header.s=20150623 header.b=tUlEGY8l; spf=neutral (google.com: 209.85.220.65 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) smtp.mailfrom=josef@toxicpanda.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id v21-v6sor59788qvf.16.2018.09.26.14.09.03 for (Google Transport Security); Wed, 26 Sep 2018 14:09:03 -0700 (PDT) Received-SPF: neutral (google.com: 209.85.220.65 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@toxicpanda-com.20150623.gappssmtp.com header.s=20150623 header.b=tUlEGY8l; spf=neutral (google.com: 209.85.220.65 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) smtp.mailfrom=josef@toxicpanda.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references; bh=FY4PvfFFgkG4rCGf8NvSi/OKKAo2xpKioHzicQvhcOI=; b=tUlEGY8l8IgAEzc/SrMWSEhdj/YyihWzeRHQm/F7LSmGImuylO2549hO9a14Megk4C S6YLhf0itc2jGfRTXuqZXNNpdLG0n/5fynFIeJx5PCGpk8ve3pr+L3HbN8JwP4oY2vNz are7A4iKZfhnLtonNzI92X7P0gMiAtPr5kJ1P1S0JSLRkB3Xy/YRf40h3yYhGpmrIPrO fINLLRqmeSOAFFhYx/4ZtRJtKCFIbY1d2A4Q0xJO0lqhVB+uTUigu5Wrre1Gp4Gbx/4Q 1/2nAOKAeZ6nAdT/3xXt7evr6JQEeAOsDYjieEmH8aGd86QB9tZKXMuOkAackq2qjcjf pGVA== X-Google-Smtp-Source: ACcGV63m+uEm9iqYhuq6Vvgdg3700+CAAqavam7D27mFj9y+w+xIfARiMWPqtlMjnrw80RLGv5qyZg== X-Received: by 2002:a0c:f906:: with SMTP id v6-v6mr5821962qvn.86.1537996143023; Wed, 26 Sep 2018 14:09:03 -0700 (PDT) Received: from localhost ([107.15.81.208]) by smtp.gmail.com with ESMTPSA id l18-v6sm96163qtb.2.2018.09.26.14.09.01 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 26 Sep 2018 14:09:02 -0700 (PDT) From: Josef Bacik To: kernel-team@fb.com, linux-kernel@vger.kernel.org, hannes@cmpxchg.org, tj@kernel.org, linux-fsdevel@vger.kernel.org, akpm@linux-foundation.org, riel@redhat.com, linux-mm@kvack.org, linux-btrfs@vger.kernel.org Subject: [PATCH 2/9] mm: drop mmap_sem for page cache read IO submission Date: Wed, 26 Sep 2018 17:08:49 -0400 Message-Id: <20180926210856.7895-3-josef@toxicpanda.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180926210856.7895-1-josef@toxicpanda.com> References: <20180926210856.7895-1-josef@toxicpanda.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Johannes Weiner Reads can take a long time, and if anybody needs to take a write lock on the mmap_sem it'll block any subsequent readers to the mmap_sem while the read is outstanding, which could cause long delays. Instead drop the mmap_sem if we do any reads at all. Signed-off-by: Johannes Weiner Signed-off-by: Josef Bacik --- mm/filemap.c | 119 ++++++++++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 90 insertions(+), 29 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 52517f28e6f4..1ed35cd99b2c 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2366,6 +2366,18 @@ generic_file_read_iter(struct kiocb *iocb, struct iov_iter *iter) EXPORT_SYMBOL(generic_file_read_iter); #ifdef CONFIG_MMU +static struct file *maybe_unlock_mmap_for_io(struct vm_area_struct *vma, int flags) +{ + if ((flags & (FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_RETRY_NOWAIT)) == FAULT_FLAG_ALLOW_RETRY) { + struct file *file; + + file = get_file(vma->vm_file); + up_read(&vma->vm_mm->mmap_sem); + return file; + } + return NULL; +} + /** * page_cache_read - adds requested page to the page cache if not already there * @file: file to read @@ -2405,23 +2417,28 @@ static int page_cache_read(struct file *file, pgoff_t offset, gfp_t gfp_mask) * Synchronous readahead happens when we don't even find * a page in the page cache at all. */ -static void do_sync_mmap_readahead(struct vm_area_struct *vma, - struct file_ra_state *ra, - struct file *file, - pgoff_t offset) +static int do_sync_mmap_readahead(struct vm_area_struct *vma, + struct file_ra_state *ra, + struct file *file, + pgoff_t offset, + int flags) { struct address_space *mapping = file->f_mapping; + struct file *fpin; /* If we don't want any read-ahead, don't bother */ if (vma->vm_flags & VM_RAND_READ) - return; + return 0; if (!ra->ra_pages) - return; + return 0; if (vma->vm_flags & VM_SEQ_READ) { + fpin = maybe_unlock_mmap_for_io(vma, flags); page_cache_sync_readahead(mapping, ra, file, offset, ra->ra_pages); - return; + if (fpin) + fput(fpin); + return fpin ? -EAGAIN : 0; } /* Avoid banging the cache line if not needed */ @@ -2433,7 +2450,9 @@ static void do_sync_mmap_readahead(struct vm_area_struct *vma, * stop bothering with read-ahead. It will only hurt. */ if (ra->mmap_miss > MMAP_LOTSAMISS) - return; + return 0; + + fpin = maybe_unlock_mmap_for_io(vma, flags); /* * mmap read-around @@ -2442,28 +2461,40 @@ static void do_sync_mmap_readahead(struct vm_area_struct *vma, ra->size = ra->ra_pages; ra->async_size = ra->ra_pages / 4; ra_submit(ra, mapping, file); + + if (fpin) + fput(fpin); + + return fpin ? -EAGAIN : 0; } /* * Asynchronous readahead happens when we find the page and PG_readahead, * so we want to possibly extend the readahead further.. */ -static void do_async_mmap_readahead(struct vm_area_struct *vma, - struct file_ra_state *ra, - struct file *file, - struct page *page, - pgoff_t offset) +static int do_async_mmap_readahead(struct vm_area_struct *vma, + struct file_ra_state *ra, + struct file *file, + struct page *page, + pgoff_t offset, + int flags) { struct address_space *mapping = file->f_mapping; + struct file *fpin; /* If we don't want any read-ahead, don't bother */ if (vma->vm_flags & VM_RAND_READ) - return; + return 0; if (ra->mmap_miss > 0) ra->mmap_miss--; - if (PageReadahead(page)) - page_cache_async_readahead(mapping, ra, file, - page, offset, ra->ra_pages); + if (!PageReadahead(page)) + return 0; + fpin = maybe_unlock_mmap_for_io(vma, flags); + page_cache_async_readahead(mapping, ra, file, + page, offset, ra->ra_pages); + if (fpin) + fput(fpin); + return fpin ? -EAGAIN : 0; } /** @@ -2479,10 +2510,8 @@ static void do_async_mmap_readahead(struct vm_area_struct *vma, * * vma->vm_mm->mmap_sem must be held on entry. * - * If our return value has VM_FAULT_RETRY set, it's because - * lock_page_or_retry() returned 0. - * The mmap_sem has usually been released in this case. - * See __lock_page_or_retry() for the exception. + * If our return value has VM_FAULT_RETRY set, the mmap_sem has + * usually been released. * * If our return value does not have VM_FAULT_RETRY set, the mmap_sem * has not been released. @@ -2492,11 +2521,13 @@ static void do_async_mmap_readahead(struct vm_area_struct *vma, vm_fault_t filemap_fault(struct vm_fault *vmf) { int error; + struct mm_struct *mm = vmf->vma->vm_mm; struct file *file = vmf->vma->vm_file; struct address_space *mapping = file->f_mapping; struct file_ra_state *ra = &file->f_ra; struct inode *inode = mapping->host; pgoff_t offset = vmf->pgoff; + int flags = vmf->flags; pgoff_t max_off; struct page *page; vm_fault_t ret = 0; @@ -2509,27 +2540,44 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) * Do we have something in the page cache already? */ page = find_get_page(mapping, offset); - if (likely(page) && !(vmf->flags & FAULT_FLAG_TRIED)) { + if (likely(page) && !(flags & FAULT_FLAG_TRIED)) { /* * We found the page, so try async readahead before * waiting for the lock. */ - do_async_mmap_readahead(vmf->vma, ra, file, page, offset); + error = do_async_mmap_readahead(vmf->vma, ra, file, page, offset, vmf->flags); + if (error == -EAGAIN) + goto out_retry_wait; } else if (!page) { /* No page in the page cache at all */ - do_sync_mmap_readahead(vmf->vma, ra, file, offset); - count_vm_event(PGMAJFAULT); - count_memcg_event_mm(vmf->vma->vm_mm, PGMAJFAULT); ret = VM_FAULT_MAJOR; + count_vm_event(PGMAJFAULT); + count_memcg_event_mm(mm, PGMAJFAULT); + error = do_sync_mmap_readahead(vmf->vma, ra, file, offset, vmf->flags); + if (error == -EAGAIN) + goto out_retry_wait; retry_find: page = find_get_page(mapping, offset); if (!page) goto no_cached_page; } - if (!lock_page_or_retry(page, vmf->vma->vm_mm, vmf->flags)) { - put_page(page); - return ret | VM_FAULT_RETRY; + if (!trylock_page(page)) { + if (flags & FAULT_FLAG_ALLOW_RETRY) { + if (flags & FAULT_FLAG_RETRY_NOWAIT) + goto out_retry; + up_read(&mm->mmap_sem); + goto out_retry_wait; + } + if (flags & FAULT_FLAG_KILLABLE) { + int ret = __lock_page_killable(page); + + if (ret) { + up_read(&mm->mmap_sem); + goto out_retry; + } + } else + __lock_page(page); } /* Did it get truncated? */ @@ -2607,6 +2655,19 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) /* Things didn't work out. Return zero to tell the mm layer so. */ shrink_readahead_size_eio(file, ra); return VM_FAULT_SIGBUS; + +out_retry_wait: + if (page) { + if (flags & FAULT_FLAG_KILLABLE) + wait_on_page_locked_killable(page); + else + wait_on_page_locked(page); + } + +out_retry: + if (page) + put_page(page); + return ret | VM_FAULT_RETRY; } EXPORT_SYMBOL(filemap_fault); From patchwork Wed Sep 26 21:08:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 10616847 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D0786174A for ; Wed, 26 Sep 2018 21:09:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C02412B774 for ; Wed, 26 Sep 2018 21:09:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B3D712B7F2; Wed, 26 Sep 2018 21:09:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 05E862B774 for ; Wed, 26 Sep 2018 21:09:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 84E7A8E0005; Wed, 26 Sep 2018 17:09:06 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7FDF28E0001; Wed, 26 Sep 2018 17:09:06 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 651A88E0005; Wed, 26 Sep 2018 17:09:06 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by kanga.kvack.org (Postfix) with ESMTP id 383B58E0001 for ; Wed, 26 Sep 2018 17:09:06 -0400 (EDT) Received: by mail-qk1-f199.google.com with SMTP id e3-v6so429698qkj.17 for ; Wed, 26 Sep 2018 14:09:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=ESNx+ZeXEpnsFX3NMmn2fGNCRcRxntWtJSga04sSIlA=; b=EAwPB8FF6B63UsQy5YtxofxrBZXNocWRrq7/8o9tu3yxv5fl0Jqt5KC+616fb0Smss gjqb8uWXao20EN7UEGB/DKbXqH2T3On3RQ6hWaw0ZrfchKk/5lwvgQQPlfhrh7AQsDBO lETGI5kPUwHwjQ43Uc1AwTe9dzH05obzlvyHET085/BwqbuhhlZ93PKs1ploZBj4fZ/h PVVnfJ9/tr919EhQJ3ph8BIZVI27ayn1XcpneqUkht0pW/Q6ZGPwS8let8C6Cc0rTz/M 2pB+F2celHWifsyxCEwMw3TNY+G/PBi1QnVpbiblcS3RSXXPfQWibbXi4WfkQ+kmRIEk p+jQ== X-Gm-Message-State: ABuFfoh0QEScijv6KriyRdZZqdRb9/IUa3hzoxK6EV6J+EcKPoud1Wa+ B0AaUpOrpgmj5kSisMjKGLBwGpRkghwg6iACopK01ge8hvOjyUbbFyKoyZJoAgS7xiDN4oz15H5 KC6/L89xX6OkkFQUExBxe8Ld9Uuo6K9wJMqAUnPIL9Rt4LoTbGsgb/wcWJ5DaXGT1hsQQurh2Vl r/pyDjf2D7fm6q8SdUD4Bz45i6/diKz13bjiENS2Nfw0s2S/WvgP7yNesmOC41hQPBKQdTdNXts LkYIjy9a11WR/ZRXDdQEFBgfU+NGpVSMej4j4lb/SPMBemb8gSGZHeDQXtaBFSGyo0UqMHUgWdj BTqe3GSDrADCmq2YUG+LCa2Vxfk/A5wgZBLSCu2ktbzBXpcqBD+GzxZsh5zxsvmEnTs2fXgqNX/ I X-Received: by 2002:a37:b101:: with SMTP id a1-v6mr5663582qkf.222.1537996145963; Wed, 26 Sep 2018 14:09:05 -0700 (PDT) X-Received: by 2002:a37:b101:: with SMTP id a1-v6mr5663532qkf.222.1537996145149; Wed, 26 Sep 2018 14:09:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537996145; cv=none; d=google.com; s=arc-20160816; b=m/Z8JYz5fbb7sQ2xiMpB7+mUnUzm6yD19gO1MEf86zutR7jsNZ2AW53OzunDprSvsa 9ackPUWGfuTaT/ElAE03NZb1TKA+Jqk6bE7WtdUfvVh1Q+31TlQizgn2gPF99cwwbomB 3Qq4oa79otHU24WtoeS6WJVKl0nYOeUdSZWdC3Aj03lT4hjb6UnFQA01Axjpt6aElGf4 Y1tXP4ptDQUH3x/BN7rp0+9N+bil4AdHELbeyUni3q1g85og6FgQvraXiNNKn39Hgb9N iS0jADoY682UKv/460FGNAZzVvGDTanAYHSd8Ucpo4IvcYcSO21zt5TYw5/qX6/7Qy1X B2BQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ESNx+ZeXEpnsFX3NMmn2fGNCRcRxntWtJSga04sSIlA=; b=GPjbWbIqNx/rdautqA59U7pqFb/QhJwF9lm5e53oMVg/FNQOZ4AcOmQzAXGWp2zP9p Nz1GVvETHOhWl9r04+zWtrIyeaJUSXMTTIcy/lBKIAOabmmCVwMCmqQKRNrQP0AchF6/ nTiZa/kOSXUXHGcibMcObcQFa0xlU771lpYDvwNXbfqi9mAmhAXDKS0h6tFTjnIXWDXJ xC6/i4VWY8jlm7eKStIAax4ejjGloXHGL7boKmQsZbAhYyyXCJFNpzOHv9AgY5zc1nxD uJUHaUU2ko47NNsmA+F+Et4RG45aJjepzqHbkzf8wYmDja8N9AuqkTBaO1Op421yRSmW A66g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@toxicpanda-com.20150623.gappssmtp.com header.s=20150623 header.b=XwT1iJ14; spf=neutral (google.com: 209.85.220.65 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) smtp.mailfrom=josef@toxicpanda.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id d18-v6sor45757qve.108.2018.09.26.14.09.05 for (Google Transport Security); Wed, 26 Sep 2018 14:09:05 -0700 (PDT) Received-SPF: neutral (google.com: 209.85.220.65 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@toxicpanda-com.20150623.gappssmtp.com header.s=20150623 header.b=XwT1iJ14; spf=neutral (google.com: 209.85.220.65 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) smtp.mailfrom=josef@toxicpanda.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ESNx+ZeXEpnsFX3NMmn2fGNCRcRxntWtJSga04sSIlA=; b=XwT1iJ14b0Y8zOY8wRLonfvQWFerAbtD/ma0JeyNfblznWcTzR6PMdo9Gxvn9vQ8Fo uJRpjsCeIPuBeJq5zpQr6RHFH6VBi0clsuwsqTKCsZbmEzxH15vrjtdd8uNMhsbvFzZf Hl8ArPBKxVjv2pIQKeQFmmkEU5eTC6L+jYdtM6fXnAMAHjQiSx39qKqFn+r0NaffcCgc PkTpgRAn0fmPpVCVO3ysC9zDbghlWIBiP7vkQyI7tP5/VAZp0Onh02N/aWIbPJXUVJND pTNKQcRXTFfCsy4cMFQUzmTIBL1Eekk3mAfbRQPKS/SRB2RZE8pGpEwD+vUVG8dtSp+h OvSA== X-Google-Smtp-Source: ACcGV612awagRXq/MVl2xwKC547NGtGmnifm3iz1tSpVjUxR1GQEkPMPqVcQoudSYXdj3D6A8K2fHg== X-Received: by 2002:a0c:9557:: with SMTP id m23-v6mr5789549qvm.138.1537996144815; Wed, 26 Sep 2018 14:09:04 -0700 (PDT) Received: from localhost ([107.15.81.208]) by smtp.gmail.com with ESMTPSA id x11-v6sm63930qtc.48.2018.09.26.14.09.03 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 26 Sep 2018 14:09:03 -0700 (PDT) From: Josef Bacik To: kernel-team@fb.com, linux-kernel@vger.kernel.org, hannes@cmpxchg.org, tj@kernel.org, linux-fsdevel@vger.kernel.org, akpm@linux-foundation.org, riel@redhat.com, linux-mm@kvack.org, linux-btrfs@vger.kernel.org Cc: Johannes Weiner Subject: [PATCH 3/9] mm: clean up swapcache lookup and creation function names Date: Wed, 26 Sep 2018 17:08:50 -0400 Message-Id: <20180926210856.7895-4-josef@toxicpanda.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180926210856.7895-1-josef@toxicpanda.com> References: <20180926210856.7895-1-josef@toxicpanda.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Johannes Weiner __read_swap_cache_async() has a misleading name. All it does is look up or create a page in swapcache; it doesn't initiate any IO. The swapcache has many parallels to the page cache, and shares naming schemes with it elsewhere. Analogous to the cache lookup and creation API, rename __read_swap_cache_async() find_or_create_swap_cache() and lookup_swap_cache() to find_swap_cache(). Signed-off-by: Johannes Weiner Signed-off-by: Josef Bacik --- include/linux/swap.h | 14 ++++++++------ mm/memory.c | 2 +- mm/shmem.c | 2 +- mm/swap_state.c | 43 ++++++++++++++++++++++--------------------- mm/zswap.c | 8 ++++---- 5 files changed, 36 insertions(+), 33 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 8e2c11e692ba..293a84c34448 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -412,15 +412,17 @@ extern void __delete_from_swap_cache(struct page *); extern void delete_from_swap_cache(struct page *); extern void free_page_and_swap_cache(struct page *); extern void free_pages_and_swap_cache(struct page **, int); -extern struct page *lookup_swap_cache(swp_entry_t entry, - struct vm_area_struct *vma, - unsigned long addr); +extern struct page *find_swap_cache(swp_entry_t entry, + struct vm_area_struct *vma, + unsigned long addr); +extern struct page *find_or_create_swap_cache(swp_entry_t entry, + gfp_t gfp_mask, + struct vm_area_struct *vma, + unsigned long addr, + bool *created); extern struct page *read_swap_cache_async(swp_entry_t, gfp_t, struct vm_area_struct *vma, unsigned long addr, bool do_poll); -extern struct page *__read_swap_cache_async(swp_entry_t, gfp_t, - struct vm_area_struct *vma, unsigned long addr, - bool *new_page_allocated); extern struct page *swap_cluster_readahead(swp_entry_t entry, gfp_t flag, struct vm_fault *vmf); extern struct page *swapin_readahead(swp_entry_t entry, gfp_t flag, diff --git a/mm/memory.c b/mm/memory.c index 433075f722ea..6f8abde84986 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2935,7 +2935,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) delayacct_set_flag(DELAYACCT_PF_SWAPIN); - page = lookup_swap_cache(entry, vma, vmf->address); + page = find_swap_cache(entry, vma, vmf->address); swapcache = page; if (!page) { diff --git a/mm/shmem.c b/mm/shmem.c index 0376c124b043..9854903ae92f 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1679,7 +1679,7 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, if (swap.val) { /* Look it up and read it in.. */ - page = lookup_swap_cache(swap, NULL, 0); + page = find_swap_cache(swap, NULL, 0); if (!page) { /* Or update major stats only when swapin succeeds?? */ if (fault_type) { diff --git a/mm/swap_state.c b/mm/swap_state.c index ecee9c6c4cc1..bae758e19f7a 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -330,8 +330,8 @@ static inline bool swap_use_vma_readahead(void) * lock getting page table operations atomic even if we drop the page * lock before returning. */ -struct page *lookup_swap_cache(swp_entry_t entry, struct vm_area_struct *vma, - unsigned long addr) +struct page *find_swap_cache(swp_entry_t entry, struct vm_area_struct *vma, + unsigned long addr) { struct page *page; @@ -374,19 +374,20 @@ struct page *lookup_swap_cache(swp_entry_t entry, struct vm_area_struct *vma, return page; } -struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, +struct page *find_or_create_swap_cache(swp_entry_t entry, gfp_t gfp_mask, struct vm_area_struct *vma, unsigned long addr, - bool *new_page_allocated) + bool *created) { struct page *found_page, *new_page = NULL; struct address_space *swapper_space = swap_address_space(entry); int err; - *new_page_allocated = false; + + *created = false; do { /* * First check the swap cache. Since this is normally - * called after lookup_swap_cache() failed, re-calling + * called after find_swap_cache() failed, re-calling * that would confuse statistics. */ found_page = find_get_page(swapper_space, swp_offset(entry)); @@ -449,7 +450,7 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, * Initiate read into locked page and return. */ lru_cache_add_anon(new_page); - *new_page_allocated = true; + *created = true; return new_page; } radix_tree_preload_end(); @@ -475,14 +476,14 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, struct page *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, struct vm_area_struct *vma, unsigned long addr, bool do_poll) { - bool page_was_allocated; - struct page *retpage = __read_swap_cache_async(entry, gfp_mask, - vma, addr, &page_was_allocated); + struct page *page; + bool created; - if (page_was_allocated) - swap_readpage(retpage, do_poll); + page = find_or_create_swap_cache(entry, gfp_mask, vma, addr, &created); + if (created) + swap_readpage(page, do_poll); - return retpage; + return page; } static unsigned int __swapin_nr_pages(unsigned long prev_offset, @@ -573,7 +574,7 @@ struct page *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, unsigned long mask; struct swap_info_struct *si = swp_swap_info(entry); struct blk_plug plug; - bool do_poll = true, page_allocated; + bool do_poll = true, created; struct vm_area_struct *vma = vmf->vma; unsigned long addr = vmf->address; @@ -593,12 +594,12 @@ struct page *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, blk_start_plug(&plug); for (offset = start_offset; offset <= end_offset ; offset++) { /* Ok, do the async read-ahead now */ - page = __read_swap_cache_async( + page = find_or_create_swap_cache( swp_entry(swp_type(entry), offset), - gfp_mask, vma, addr, &page_allocated); + gfp_mask, vma, addr, &created); if (!page) continue; - if (page_allocated) { + if (created) { swap_readpage(page, false); if (offset != entry_offset) { SetPageReadahead(page); @@ -738,7 +739,7 @@ static struct page *swap_vma_readahead(swp_entry_t fentry, gfp_t gfp_mask, pte_t *pte, pentry; swp_entry_t entry; unsigned int i; - bool page_allocated; + bool created; struct vma_swap_readahead ra_info = {0,}; swap_ra_info(vmf, &ra_info); @@ -756,11 +757,11 @@ static struct page *swap_vma_readahead(swp_entry_t fentry, gfp_t gfp_mask, entry = pte_to_swp_entry(pentry); if (unlikely(non_swap_entry(entry))) continue; - page = __read_swap_cache_async(entry, gfp_mask, vma, - vmf->address, &page_allocated); + page = find_or_create_swap_cache(entry, gfp_mask, vma, + vmf->address, &created); if (!page) continue; - if (page_allocated) { + if (created) { swap_readpage(page, false); if (i != ra_info.offset) { SetPageReadahead(page); diff --git a/mm/zswap.c b/mm/zswap.c index cd91fd9d96b8..6f05faa75766 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -823,11 +823,11 @@ enum zswap_get_swap_ret { static int zswap_get_swap_cache_page(swp_entry_t entry, struct page **retpage) { - bool page_was_allocated; + bool created; - *retpage = __read_swap_cache_async(entry, GFP_KERNEL, - NULL, 0, &page_was_allocated); - if (page_was_allocated) + *retpage = find_or_create_swap_cache(entry, GFP_KERNEL, + NULL, 0, &created); + if (created) return ZSWAP_SWAPCACHE_NEW; if (!*retpage) return ZSWAP_SWAPCACHE_FAIL; From patchwork Wed Sep 26 21:08:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 10616851 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 16D2A15A6 for ; Wed, 26 Sep 2018 21:09:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 094952B7F2 for ; Wed, 26 Sep 2018 21:09:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F0CAC2B7F9; Wed, 26 Sep 2018 21:09:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A0E2B2B7F2 for ; Wed, 26 Sep 2018 21:09:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DF9AE8E0006; Wed, 26 Sep 2018 17:09:07 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D7B028E0001; Wed, 26 Sep 2018 17:09:07 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C488B8E0007; Wed, 26 Sep 2018 17:09:07 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by kanga.kvack.org (Postfix) with ESMTP id 94ED48E0006 for ; Wed, 26 Sep 2018 17:09:07 -0400 (EDT) Received: by mail-qk1-f200.google.com with SMTP id p192-v6so449097qke.13 for ; Wed, 26 Sep 2018 14:09:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=L4OEsFz3WUjgEGzu73gT/wwAqaCABq/L1QMv8mr7jYM=; b=IsuX/n/5TwzX8RzDx7OnhkB3Lg/BO5DAMMgL1fWOlWPQ/I3wPijdqYx4nV+zy/z7or kp53oq0lQrDuJpQJ/oGkM01Iwd3dVgkn+a62AoHyE9GMTFCFRspc3Pi9gqEliB8Yynde UTwsDynzG6reCDFt31srRCCEm2XmN9WU1HxhtXZj14n3/8uot4o5wSR6+1C/y3MY9uqZ 51rdj8KB7dHFe498Y5w0F7p7/CpRdqWCWDRmjcLmNJKMRZBq9LVFsDzlk7cp7F2Q5+Qc qUzi9b5wh3FvTmMNG1vqvenV9vogymLGis//fVwIhepIWsdh4/gFC9UQwSYyQmMImVGB 4LSw== X-Gm-Message-State: ABuFfohZN4zOSpznrGp0kT/oq1S2ALTS8nT5KIN9vvVfOLp0TW17uVEz PrtWFyQFQlxIYSnaWz87q15g3d8biPpBVeN7g2BLqly3pgf707UHJVORnVGRzWEyTG8//Eo/ocs SR6wo9KYO2NUm+D0ANt1l0f4FyoSGqwQ4qoP5l29u2cX0XbvqP1e538oL6TmrjG05Y62uLV8SWp VAOzWqwVwITDA0o3ClplPnWjBIiMH+VRywhw+X9JoZmHYNojJOB1REQHSMroCZEnw/VThRW02vL oD7ndjuDitV4Ir7cCxVpe19mlUeOahzurtp6o4ghFjz5Ww0XBNC3D7QuOy3khzsvjmKZPKp45gY IPRHUbQ0BTeg+xFfRLSbhGCWyOxMwuHNV7AmZN2tlixMJdk+fzMhGaI7twbUTSw20DTOUG10mg8 F X-Received: by 2002:ac8:3882:: with SMTP id f2-v6mr6075662qtc.337.1537996147400; Wed, 26 Sep 2018 14:09:07 -0700 (PDT) X-Received: by 2002:ac8:3882:: with SMTP id f2-v6mr6075628qtc.337.1537996146844; Wed, 26 Sep 2018 14:09:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537996146; cv=none; d=google.com; s=arc-20160816; b=hBhlcjGuClp9969siD6P/zGbh/xAn9yk1GisrIyWdYn/XKJZcUbMZKTypI4VjxkZKw 77tMVrTfop8FYicKc24eVvylhUESc1eyl7UVwDujGjr19v5cF9e7JiHq93i3w/E1ABQK pO5N2GQ8rFJVRnmx9NoJErDNqP3HqpDq9mwi6diu6EaPiENeqBxAp29cwqNNsUOtxdFz suPaAKk3oB7ZMx8taD5qA8losqwbShuxokWoyRt2Jv4EMe2aWBj363jWZ4T943wFa15f d5D/K/mu7aIX/CBHCcPq8A5nRTNgilJDXQE4Y+xVwo7pJPSPkmIm+4SEjbD2birtsXc6 HplQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=L4OEsFz3WUjgEGzu73gT/wwAqaCABq/L1QMv8mr7jYM=; b=ECzbSByFzGGadGCFpbxfX4tWphj7ixsuvCJjlfOLLebVsiSzO71/K/oPoYc8rDorKL d6Ka1VpC2/iS8LpkU2prbeG4Bwa22PhTE6Ba+j/dTearoimSo6VIQCUtvQaf9kUzoJm1 KoT2nTNWM6EHJmmShASnLfWlRT692YIBnhjJrcjXXqE+HE3r8Wo+KxruR6QD9k7dTP82 kqqUTLwsV8ZIexRb4JsiWSV779ib6Wfu9+sVL6MEFZ6ARthncTkEzologKVka/qIlDMk RCkSxXbMC1PebaqMQtgBgwFrylxKb6qgpZakbnahpt7WVox2aXxGJ04raCr2+MHFe2aS G1fQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@toxicpanda-com.20150623.gappssmtp.com header.s=20150623 header.b=KzuvBbua; spf=neutral (google.com: 209.85.220.65 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) smtp.mailfrom=josef@toxicpanda.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id s15-v6sor47035qvm.105.2018.09.26.14.09.06 for (Google Transport Security); Wed, 26 Sep 2018 14:09:06 -0700 (PDT) Received-SPF: neutral (google.com: 209.85.220.65 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@toxicpanda-com.20150623.gappssmtp.com header.s=20150623 header.b=KzuvBbua; spf=neutral (google.com: 209.85.220.65 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) smtp.mailfrom=josef@toxicpanda.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=L4OEsFz3WUjgEGzu73gT/wwAqaCABq/L1QMv8mr7jYM=; b=KzuvBbuaLur8lPktdpvFa9kCRzdghPnke7Nb7qsvyTs/r/mFVoWLTwlp7SbvK6dod8 h27MFYZuD9ST4VbzRwOEKAPER5XjWL9YfkQPGPGFFKJJ6eZmg02p6dUFJwEGkOo0Rp/o 2s+oI3lPekVrwV9EYh0RZoBwGzue0DgWZSj+x5BHKmtXeppLEZBHNBbQx5h4xFeThEDS Wmq33Kk99E24OGVVW9Sd2miTsmwdg2muGlBP6bfTZMxpMudzwt1BTKmf3v+h4Wh1UIML UqAzfJT2d+TRIzWYbwEfQxFoEoEYoAlW1oHwwHLIcspURIeynYfIPaJNnSCLoD45CrDD LZXQ== X-Google-Smtp-Source: ACcGV634tXLL0I1iexAP2WhHj2bvmdrFTXsl3OPvRmlNb/A+m920mCwkcsQS0SkprH9fcqZ0jS0BqQ== X-Received: by 2002:a0c:f906:: with SMTP id v6-v6mr5822169qvn.86.1537996146591; Wed, 26 Sep 2018 14:09:06 -0700 (PDT) Received: from localhost ([107.15.81.208]) by smtp.gmail.com with ESMTPSA id n25-v6sm87327qkl.8.2018.09.26.14.09.05 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 26 Sep 2018 14:09:05 -0700 (PDT) From: Josef Bacik To: kernel-team@fb.com, linux-kernel@vger.kernel.org, hannes@cmpxchg.org, tj@kernel.org, linux-fsdevel@vger.kernel.org, akpm@linux-foundation.org, riel@redhat.com, linux-mm@kvack.org, linux-btrfs@vger.kernel.org Cc: Johannes Weiner Subject: [PATCH 4/9] mm: drop mmap_sem for swap read IO submission Date: Wed, 26 Sep 2018 17:08:51 -0400 Message-Id: <20180926210856.7895-5-josef@toxicpanda.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180926210856.7895-1-josef@toxicpanda.com> References: <20180926210856.7895-1-josef@toxicpanda.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Johannes Weiner We don't need to hold the mmap_sem while we're doing the IO, simply drop it and retry appropriately. Signed-off-by: Johannes Weiner Signed-off-by: Josef Bacik --- mm/page_io.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/mm/page_io.c b/mm/page_io.c index aafd19ec1db4..bf21b56a964e 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -365,6 +365,20 @@ int swap_readpage(struct page *page, bool synchronous) goto out; } + /* + * XXX: + * + * Propagate mm->mmap_sem into this function. Then: + * + * get_file(sis->swap_file) + * up_read(mm->mmap_sem) + * submit io request + * fput + * + * After mmap_sem is dropped, sis is no longer valid. Go + * through swap_file->blah->bdev. + */ + if (sis->flags & SWP_FILE) { struct file *swap_file = sis->swap_file; struct address_space *mapping = swap_file->f_mapping; From patchwork Wed Sep 26 21:08:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 10616855 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 685D7180E for ; Wed, 26 Sep 2018 21:09:19 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5B4252B7F2 for ; Wed, 26 Sep 2018 21:09:19 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4BDF62B80C; Wed, 26 Sep 2018 21:09:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DDE292B7F8 for ; Wed, 26 Sep 2018 21:09:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1E9F28E0007; Wed, 26 Sep 2018 17:09:10 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 14C9A8E0008; Wed, 26 Sep 2018 17:09:10 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F04D68E0007; Wed, 26 Sep 2018 17:09:09 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by kanga.kvack.org (Postfix) with ESMTP id C438F8E0001 for ; Wed, 26 Sep 2018 17:09:09 -0400 (EDT) Received: by mail-qt1-f200.google.com with SMTP id k10-v6so366561qtb.8 for ; Wed, 26 Sep 2018 14:09:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:subject:date:message-id :in-reply-to:references; bh=fxF9egp/0MfNC6eP3oXuHL949t8miVwsCl5P35THCQ0=; b=tZzdH+FWd9lzjwrvnwa9Xc5Mu8mhDUIIIJ2X3CNpCsyfRi6qXu3ecUsWx1XpCzbG31 DaqyA7cbEulynqXqkByoEK+PUJjKrM2Jsc+JIDLk43Ha953vVCuTFgaWfNpx+Gvl9am3 KA9Yvhje1O3IZzhHSPWfPsbTc/g9nkTXYRcXvG8zMJjmgxqmPeodklLBINK+pb+g9IX2 ufwDBEmQtR4JMKr4Vkw2h4/SWAoKW5gflwkC0Gv3KjkLf+sj32TAaI5GAhQmrHgFvW9Y w6jGVJNllVGEDMr2dWDjifieOCDCuWoZEJzNOZWIX/nOZSohpK6r+S1FAVtZmHjYxHvR QmnA== X-Gm-Message-State: ABuFfohVdhAHR7dsMn9aVEdOqFRTOAsmTvsPbt+JhvXSoR44EkytCmZd znYxexn/jKR9jti3XTeOWUmMgHRaHpeEuWtSHZFgnij1miJnp0Bfd0ljSf12XzXp9kqgUZYPoLU yMgok7pjIiVuLw8C0yMVQpmBbOw9wrXoS9CVx5JjIFPWnmp8kdMhckha83PqJWEvbYrcucpRI6M b6I+HguM2kqCPzcvRCHqRdMOSrpu7BBcXlujzDJS3IQuLqYkB6ItVI3TheG3UHTzFs/BWQBoMFP zqN+A3jWVLnnadiA52uFGRykgCst6k0WO6Dq5C0cSDanqeDtYla6+jVeCYo8wLrqA2o/x8MD3qS ObL1Mqj9uW/RAz80IRa+CZh/9EqNmrFRS54Ak9zgTtK76e4tCWqvObL3QJnPMgurEk0rkj4askK 0 X-Received: by 2002:a37:9f85:: with SMTP id i127-v6mr5753965qke.113.1537996149594; Wed, 26 Sep 2018 14:09:09 -0700 (PDT) X-Received: by 2002:a37:9f85:: with SMTP id i127-v6mr5753921qke.113.1537996148645; Wed, 26 Sep 2018 14:09:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537996148; cv=none; d=google.com; s=arc-20160816; b=LyVRvlF1WRAd4Gorl/ZgsEASj8kYuzrY7V+ASg15tm5vuPjHO7dLhg7A0+yQ4JA+Rz SBgPuLVsrUagd6g3172yQy/Gl9/dOQ1Gx6+h54OP31aPXLdeIu7YkJXHUrVp8l/Vqhki tw9237sN/zY2FQ8ytVFzBFQYNhbsj7kScYasrkItZbvevnEDgXVUT5NawCwJDm+JIGnj TBwVZDZQknJA0TTr0AWiZJvcDGj9S4PXjSgNvAiF967HIOkoL9bwRPNzPq69kvPPpgMD twMI3g9osO/XThO8CTIc6IC9Y3CshMv9hYYOon5/fz+fiKALX7Nz9ET3WU0+oAJx2qWe 42dQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=fxF9egp/0MfNC6eP3oXuHL949t8miVwsCl5P35THCQ0=; b=ccqPgEc/Vhnkr5QjqAoxRky052CCdC5ddldqZ3M0SFbsK288ZqT+EhUJRaFeU2Uhes 6Wwi5ZsHUu8E/ZSHilXELTQTybNBjmV4+fJDqn6znf9b4vs6s/ZiUS8UiivNZGXGbsm3 RGAzSJFpQzVCUqaz5KMm80mk2jvcg7e/2Yf9sZ09i/siDGdL+kSRnGi2C5FtBhChLEx9 qR0jEDeM4Qki0dqljufoPPcr23IBKvs2AY10a462cv4cs2ynpf3xRmBXpJZreQ7qPUUY 38BQwfKNac2n0/AgUTqHX4xW1uoONlhsZLocOYt9TZgi1Kz7oxLEXQpp5VpqhwibDP+k F+zw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@toxicpanda-com.20150623.gappssmtp.com header.s=20150623 header.b=xOG6WUhz; spf=neutral (google.com: 209.85.220.65 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) smtp.mailfrom=josef@toxicpanda.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id z204-v6sor56538qka.18.2018.09.26.14.09.08 for (Google Transport Security); Wed, 26 Sep 2018 14:09:08 -0700 (PDT) Received-SPF: neutral (google.com: 209.85.220.65 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@toxicpanda-com.20150623.gappssmtp.com header.s=20150623 header.b=xOG6WUhz; spf=neutral (google.com: 209.85.220.65 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) smtp.mailfrom=josef@toxicpanda.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references; bh=fxF9egp/0MfNC6eP3oXuHL949t8miVwsCl5P35THCQ0=; b=xOG6WUhzNHnJDZ8nzE96i5QNtT6Li8Q5zD7JZrUCY2qx9cUIzLHEP5+zgNwBSUtFyp Vsfjd4p80uZmD0X8I4mjGdoXPyZvnafq7f/RBkRbGjEddLbwCS/WwDEsPVkieZMO3WT7 9GYB6qMA/ROB+5KW8+jhZX3ScR0o+/oJHMadB8CL19veYAyrsYcy3UDgvQdXAdmVLpkp 3Lih8/OrYQEWboL0v9vg8SCRCFeaSEwiqoZebAIL5B0MyFM2arHx3endVqpa2rnLVSFF n40P6Ou4BESYXohRryM+Ny3UOyY2G+mJqJ3eWtgKvm8hpnmXLoKe2UZSeAxwErLr5ICk dqww== X-Google-Smtp-Source: ACcGV62wZZcLGsIgIA0AgxrBDu/rXm6AEl8TqHyyC/dL/kabdkKJb3x4NckjbVpljlp6VQXrWxsz0A== X-Received: by 2002:ae9:e892:: with SMTP id a140-v6mr5494846qkg.242.1537996148385; Wed, 26 Sep 2018 14:09:08 -0700 (PDT) Received: from localhost ([107.15.81.208]) by smtp.gmail.com with ESMTPSA id o7-v6sm50246qkc.67.2018.09.26.14.09.07 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 26 Sep 2018 14:09:07 -0700 (PDT) From: Josef Bacik To: kernel-team@fb.com, linux-kernel@vger.kernel.org, hannes@cmpxchg.org, tj@kernel.org, linux-fsdevel@vger.kernel.org, akpm@linux-foundation.org, riel@redhat.com, linux-mm@kvack.org, linux-btrfs@vger.kernel.org Subject: [PATCH 5/9] mm: drop the mmap_sem in all read fault cases Date: Wed, 26 Sep 2018 17:08:52 -0400 Message-Id: <20180926210856.7895-6-josef@toxicpanda.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180926210856.7895-1-josef@toxicpanda.com> References: <20180926210856.7895-1-josef@toxicpanda.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Johannes' patches didn't quite cover all of the IO cases that we need to drop the mmap_sem for, this patch covers the rest of them. Signed-off-by: Josef Bacik --- mm/filemap.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/mm/filemap.c b/mm/filemap.c index 1ed35cd99b2c..65395ee132a0 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2523,6 +2523,7 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) int error; struct mm_struct *mm = vmf->vma->vm_mm; struct file *file = vmf->vma->vm_file; + struct file *fpin = NULL; struct address_space *mapping = file->f_mapping; struct file_ra_state *ra = &file->f_ra; struct inode *inode = mapping->host; @@ -2610,11 +2611,15 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) return ret | VM_FAULT_LOCKED; no_cached_page: + fpin = maybe_unlock_mmap_for_io(vmf->vma, vmf->flags); + /* * We're only likely to ever get here if MADV_RANDOM is in * effect. */ error = page_cache_read(file, offset, vmf->gfp_mask); + if (fpin) + goto out_retry; /* * The page we want has now been added to the page cache. @@ -2634,6 +2639,8 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) return VM_FAULT_SIGBUS; page_not_uptodate: + fpin = maybe_unlock_mmap_for_io(vmf->vma, vmf->flags); + /* * Umm, take care of errors if the page isn't up-to-date. * Try to re-read it _once_. We do this synchronously, @@ -2647,6 +2654,8 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) if (!PageUptodate(page)) error = -EIO; } + if (fpin) + goto out_retry; put_page(page); if (!error || error == AOP_TRUNCATED_PAGE) @@ -2665,6 +2674,8 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) } out_retry: + if (fpin) + fput(fpin); if (page) put_page(page); return ret | VM_FAULT_RETRY; From patchwork Wed Sep 26 21:08:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 10616857 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2197A3CF1 for ; Wed, 26 Sep 2018 21:09:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0E2CC2B7F2 for ; Wed, 26 Sep 2018 21:09:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 025002B7F8; Wed, 26 Sep 2018 21:09:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 84B152B7FC for ; Wed, 26 Sep 2018 21:09:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6A0A48E0001; Wed, 26 Sep 2018 17:09:11 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6292F8E0008; Wed, 26 Sep 2018 17:09:11 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 51D318E0001; Wed, 26 Sep 2018 17:09:11 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by kanga.kvack.org (Postfix) with ESMTP id 235D28E0008 for ; Wed, 26 Sep 2018 17:09:11 -0400 (EDT) Received: by mail-qt1-f197.google.com with SMTP id d12-v6so358901qtk.13 for ; Wed, 26 Sep 2018 14:09:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:subject:date:message-id :in-reply-to:references; bh=hGTHjPlv3HOhn5TB2Ipo9k7piqoFCis94sG3VyiCA5k=; b=N0retUaey2Uezq3zIg1xlSMC2bO5x9Jzm5awr9HxXHs1nT/d2kAXbV9u/FhS/FAsZ4 LGAmGGVAqUoQ5+vsDqSM7RVmDibxwATyzbqRNOJJ5VIByQ/1NvVWmyAvnxPeXTAGhlBE HB9HyMvjbGD6z2RD4RMGS/kjPhn+uxGnAr4z8WwNej5JT6x74A1tnvBpfN0xm2M/FY6x mdrL06vAgCCvsGIcJVSGoiPty0LdLccFbYa15LCmVba4BaDd2l5EDkG8l7RQS7K9NLgz qWbb00zWIwRLg6VQ5zfCuUORQnUDeLINFEK3GhxMmvLaz8jlcZNOQ8uQ2Sjci2WXXLs/ gMJg== X-Gm-Message-State: ABuFfojLXgzdxXbxwfBXgUZY5hq8jGF2AYY87/m/3l/xWMwoyHQu8Pwk S7F88igdD84gQ4pVtMBj0UYgmEOkOQkKb2iKzACOhuxFCAspb6ePh0tBrXFvqPjh6cCnLNkFQN1 YkwS/YDdbfOkAACwgCh27Cu6OdlJixCHc/SjQtDGnPpmV55LWKQCC8WKGyOSz+EpIZcIZ7KnzSA fx8+xNLllCKh9omGDUPZCgIL8kmm+ce1ZSiiH/fON+3y7gAuLxCYCJgHDVDbeSajY2NiS5N+Esu Nsrk06lxoF6zcryi7fqTLEmq49tY9j8OLVOEu/Y1Wam4V3g1BRsU0wVo72n3rgyi361cEDHfLO2 uRADoi0g6hsSPPwFMoDR5AVyanL7JxM78dZFlOcIy4OjeialPGQFvv+eC5qwWPszXFNPlWeGgom / X-Received: by 2002:aed:2050:: with SMTP id 74-v6mr5992336qta.166.1537996150922; Wed, 26 Sep 2018 14:09:10 -0700 (PDT) X-Received: by 2002:aed:2050:: with SMTP id 74-v6mr5992303qta.166.1537996150345; Wed, 26 Sep 2018 14:09:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537996150; cv=none; d=google.com; s=arc-20160816; b=crJqnaAB+e0JsJg4Y2hrDbT3UGrXRmTOOPv1SBACNlFHS+CK28Wxdt8yk1QK1yOEgA MAvvkyMdaET4dkbP6n+7QbgWc6wbrBJf9W1Ka1mMVlb01RxpA3GdJu1c9AIgzc4WzRet bNwAFs7Oac22gW1enGhEuXWMG56pnCYTYjCRLpFVMuD/MTjSuGR5KFK9swUUXZkyHOyh zBdt+pVkc7N8TsdAWQYuC/MwH3ooK6l6qcs8pRVGGjVFb/thTVQYVoAWYBOUqL2TDRpJ QI90omlYpe4sDLZip1fEvQDjuVFGMioUiQtunmkha0SOwamm/cSQyDRIyBWCYRkuKnAw QxEQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=hGTHjPlv3HOhn5TB2Ipo9k7piqoFCis94sG3VyiCA5k=; b=0U7eUyWk2evSzYjNHj+qAbOVz0Qr23rM0//ZYd+fFvLMXG4CWVgPtNWkXgfjvoVBMR Y6UPMEFfQPeb7LA4VEx9fvv6+LdcCwPwBNjcVow3ZLPeCrqr5+E3vVstbeEucyaCr0U9 +zd1TecJ/8RjUMWeYXYxe4xtdEw8otAp3zv/Q62HBZZQxPUBe8cGICIkWD2E1EWhiOTj wbQXo90/R96R3ucS2kj8uzZrV+fKU6dX7C+RDjLVBcaZ9ROAaSN6NmgzO7L2wV+KFaeB Jnyr0HtZlFpIrF3dgVIp4rVlRvTvX+ZZtl4cbrjra6CpnEi+5gYnweOZbSiqEdnRhtUW C7mw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@toxicpanda-com.20150623.gappssmtp.com header.s=20150623 header.b="NSXsHzI/"; spf=neutral (google.com: 209.85.220.41 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) smtp.mailfrom=josef@toxicpanda.com Received: from mail-sor-f41.google.com (mail-sor-f41.google.com. [209.85.220.41]) by mx.google.com with SMTPS id z47-v6sor64108qtg.10.2018.09.26.14.09.10 for (Google Transport Security); Wed, 26 Sep 2018 14:09:10 -0700 (PDT) Received-SPF: neutral (google.com: 209.85.220.41 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) client-ip=209.85.220.41; Authentication-Results: mx.google.com; dkim=pass header.i=@toxicpanda-com.20150623.gappssmtp.com header.s=20150623 header.b="NSXsHzI/"; spf=neutral (google.com: 209.85.220.41 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) smtp.mailfrom=josef@toxicpanda.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references; bh=hGTHjPlv3HOhn5TB2Ipo9k7piqoFCis94sG3VyiCA5k=; b=NSXsHzI/XYShoNL8eGn0laEEoasfKJMhCPMQwHJ2Yvq8bwk04MyKO9oggPZ3BkE18j olp6YBCPQkeZXNp6oFQ9wE8pCqCakgzN1+c7Yf+qmYZnyq8kxdmoMhi8XJ6C2WRQrGQy nRoPCsn5CexUjHhSeDaxBU9NGB0VSELXL9DK0C6Xgh9Ttpe1SYyyAjXUFByg3xAEnsQR o3wyXeLFBnMiBJl6O7+7dZtJdtDKFbvgEAqm2FnuF2w27uQ/bdPH0KQypM/fZS0r/kzB YsM3KwlXFUWsqcyPrPhoIeoekAN5UBtpHcpIjSVtC5HZe9iiYCCrMpVAVC43s00HglYC UBCQ== X-Google-Smtp-Source: ACcGV63X6GJlDkWL3trIfKsJDYg6xMkxeDmRSldNmGu6BvcZd3OT+4MyGR7tfq+GMXsjHNWbVQwYBQ== X-Received: by 2002:ac8:2672:: with SMTP id v47-v6mr5865520qtv.287.1537996150087; Wed, 26 Sep 2018 14:09:10 -0700 (PDT) Received: from localhost ([107.15.81.208]) by smtp.gmail.com with ESMTPSA id y143-v6sm33940qky.95.2018.09.26.14.09.09 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 26 Sep 2018 14:09:09 -0700 (PDT) From: Josef Bacik To: kernel-team@fb.com, linux-kernel@vger.kernel.org, hannes@cmpxchg.org, tj@kernel.org, linux-fsdevel@vger.kernel.org, akpm@linux-foundation.org, riel@redhat.com, linux-mm@kvack.org, linux-btrfs@vger.kernel.org Subject: [PATCH 6/9] mm: use the cached page for filemap_fault Date: Wed, 26 Sep 2018 17:08:53 -0400 Message-Id: <20180926210856.7895-7-josef@toxicpanda.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180926210856.7895-1-josef@toxicpanda.com> References: <20180926210856.7895-1-josef@toxicpanda.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP If we drop the mmap_sem we have to redo the vma lookup which requires redoing the fault handler. Chances are we will just come back to the same page, so save this page in our vmf->cached_page and reuse it in the next loop through the fault handler. Signed-off-by: Josef Bacik --- mm/filemap.c | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 65395ee132a0..5212ab637832 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2530,13 +2530,38 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) pgoff_t offset = vmf->pgoff; int flags = vmf->flags; pgoff_t max_off; - struct page *page; + struct page *page = NULL; + struct page *cached_page = vmf->cached_page; vm_fault_t ret = 0; max_off = DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE); if (unlikely(offset >= max_off)) return VM_FAULT_SIGBUS; + /* + * We may have read in the page already and have a page from an earlier + * loop. If so we need to see if this page is still valid, and if not + * do the whole dance over again. + */ + if (cached_page) { + if (flags & FAULT_FLAG_KILLABLE) { + error = lock_page_killable(cached_page); + if (error) { + up_read(&mm->mmap_sem); + goto out_retry; + } + } else + lock_page(cached_page); + vmf->cached_page = NULL; + if (cached_page->mapping == mapping && + cached_page->index == offset) { + page = cached_page; + goto have_cached_page; + } + unlock_page(cached_page); + put_page(cached_page); + } + /* * Do we have something in the page cache already? */ @@ -2587,6 +2612,7 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) put_page(page); goto retry_find; } +have_cached_page: VM_BUG_ON_PAGE(page->index != offset, page); /* @@ -2677,7 +2703,7 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) if (fpin) fput(fpin); if (page) - put_page(page); + vmf->cached_page = page; return ret | VM_FAULT_RETRY; } EXPORT_SYMBOL(filemap_fault); From patchwork Wed Sep 26 21:08:54 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 10616861 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D02C615A6 for ; Wed, 26 Sep 2018 21:09:26 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BF2B22B7F2 for ; Wed, 26 Sep 2018 21:09:26 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B346B2B7F9; Wed, 26 Sep 2018 21:09:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 493302B7F2 for ; Wed, 26 Sep 2018 21:09:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3934C8E0009; Wed, 26 Sep 2018 17:09:13 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 343318E0008; Wed, 26 Sep 2018 17:09:13 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 20B3E8E0009; Wed, 26 Sep 2018 17:09:13 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by kanga.kvack.org (Postfix) with ESMTP id E9C768E0008 for ; Wed, 26 Sep 2018 17:09:12 -0400 (EDT) Received: by mail-qk1-f199.google.com with SMTP id s22-v6so475509qkg.2 for ; Wed, 26 Sep 2018 14:09:12 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:subject:date:message-id :in-reply-to:references; bh=rRSaxONrZWxSfcObSyXaZynV5IrpMUJ0BUHUIDN5YaE=; b=mqq5/2smhunaA3fk7Eldrt0CE8M03dchVuCK8kbTy1y9FiMWJFAUxB8A6F0et0WEI+ +VZcboQWkzU2kTbvpbtX9W+qUm7QbccaWQGpnsj6PEGTfbm2+hzBXAclZwFD2WPscz/k S09OEMD6QIvAq189Q2MlmpUJSXosZY/n2kVn2xVAdaQzw4J95FMKptTVk/YUmyRbpDsn rfWbkUaorHEySxyxhfQQa600s0kypFGcmnvKKyQDrLW18+wqbKzzRdi8DLtO3V0ND3BW TLSQ6x3ikJVSZN6pMdg9W5SlJvjWhsJZXk6LjkK3JFc62uqPgQwrUNpZ4G7Kn57u7yvy 8ViQ== X-Gm-Message-State: ABuFfohYV1e2EBgAFejR18+VS5K+vvJbUxSZOrJO3lStALoyUG1Svj6m Sd9OpruHDQZZ5+YuFMK/AIRruFCcql8rupSNlG6m+TMEwLvx6nCa85wkdOCvY7Dtu+kawFzhgXt m8e5fzlk457bxWCu12l0fsgS2WTG27gXu0Ek4gbsEP9woBRDX0i0uBfPq918PpM7oEDThrQ0v5D YXUOQ6w5NGG7oaDDbaydOwjyysjRz+zMj38wvDTi0ENO97z/U4mhPxZnIB40Qrlv1JJj0DhjMJf aPI3INGKNpIsEjvDmXQumxYkHJgmCDaHIGYaUD+UQXo0mNynHwWb3RStZLwGIucxQ7V6cpPcs4N RHpp7QfC9LSofRH5oPxeyItNu4QX7zrSxeTLSiAFyWnR0+GKitTMUKhurO74nOkvkNnepTrF+l4 0 X-Received: by 2002:a0c:cdc8:: with SMTP id a8-v6mr5773310qvn.14.1537996152722; Wed, 26 Sep 2018 14:09:12 -0700 (PDT) X-Received: by 2002:a0c:cdc8:: with SMTP id a8-v6mr5773284qvn.14.1537996152130; Wed, 26 Sep 2018 14:09:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537996152; cv=none; d=google.com; s=arc-20160816; b=D7xcvfzpPt4eKILPeX44kWi7hdSRWXYoCYy5ZtreX1KRE4ei3SREGoofZXW9wbWrbi Lne8l32EIAKS8fJCCZXozAWLlqD0JTYXLm/G4uXVynag5liz/fqwlNQ6ORr/Xv1Ls0il f1ZFEBLirJZrjetBv9TD53ibl65dc2+86U/HCLIRYNiuP5Ddmkql0GX+hdKoX7xkAF/1 I3u24MjBsSK1rz8oXn0jyV24I9aLyw6yarAhOzSjbx2fLDDZPeOlFN3k9uzJP9OGCCO0 BI3gM6QOm02m7T6DcoLFjIUK+IhlY2Lgz2Q1YvqXdLr6I5ZZeI2T+SM3c/+5tlo+ZmUT ifOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=rRSaxONrZWxSfcObSyXaZynV5IrpMUJ0BUHUIDN5YaE=; b=OiuE0OEvCs3PsZkgYrlwEYgFkvA6Op3Tn76rFsD1XshEwPgNRL3FqXLwwZFXAYcKRO FPdwaMAfujaebyavTYJeYpc4luRdcJVk2EM6oFORN3WRTBM7ulO9Hjg2cBQYnM0WefLj JKk17BB2awhVpfT3zJg4yCa7KqQ+iQHfHSteod1ZjUloWpPStmUSf2gpK48srifyW8CT lgFOFlfYrU60ko/lUK283mkbaFWjBx2fWhMWo4QFUyZR7Id+m41Pf+FSTqMA3iR4Gsli UMjtuVI9W9cjdkXpJNvuV16XA6MQCQSHmrxFl81yFms4WewJ7AHUpjsSwxcfdu6Taz+4 feFw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@toxicpanda-com.20150623.gappssmtp.com header.s=20150623 header.b=cMCnruOr; spf=neutral (google.com: 209.85.220.65 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) smtp.mailfrom=josef@toxicpanda.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id k22-v6sor44957qtq.95.2018.09.26.14.09.12 for (Google Transport Security); Wed, 26 Sep 2018 14:09:12 -0700 (PDT) Received-SPF: neutral (google.com: 209.85.220.65 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@toxicpanda-com.20150623.gappssmtp.com header.s=20150623 header.b=cMCnruOr; spf=neutral (google.com: 209.85.220.65 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) smtp.mailfrom=josef@toxicpanda.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references; bh=rRSaxONrZWxSfcObSyXaZynV5IrpMUJ0BUHUIDN5YaE=; b=cMCnruOrfxbswXpQ+LzDNW4YBVLUWDQs7bq7X5QQbmFe7IBXo+BpnpXZ5nYL+NtYUu UmQOdRfXvT1X04/qw5qRpxkrQHPA9jdosXRIjLUl63CsIpykwMRlwwk/l9e21291y+Il RTMGa1o+lcurVhm+IrAgprHHyRMmR9Q34XHI1p9+jyEySWWfd0/sGQRtQ0sm1r1/OTYR Jn4qDuTHXr1vCpWOEV5xBltDo+2f4ntWFKOsFfmgVmZN08cvjpw0dJcEIgDceEBJqAxt /zQ0Z7X8q4v+RVYpOm8Cnq8jVqw7E/fQDxH+bhVROQZiPF3i35ff81yOhbZzjGw05Nmr vLBQ== X-Google-Smtp-Source: ACcGV63nJY6gaLc/Nzii08FEPUVIkCZuTe+OVlkmq03gRhXyS5p+AaMCVfImTqMIHfOrrPvVJ2Pfgg== X-Received: by 2002:ac8:2a81:: with SMTP id b1-v6mr6223154qta.286.1537996151881; Wed, 26 Sep 2018 14:09:11 -0700 (PDT) Received: from localhost ([107.15.81.208]) by smtp.gmail.com with ESMTPSA id u3-v6sm59038qku.46.2018.09.26.14.09.10 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 26 Sep 2018 14:09:10 -0700 (PDT) From: Josef Bacik To: kernel-team@fb.com, linux-kernel@vger.kernel.org, hannes@cmpxchg.org, tj@kernel.org, linux-fsdevel@vger.kernel.org, akpm@linux-foundation.org, riel@redhat.com, linux-mm@kvack.org, linux-btrfs@vger.kernel.org Subject: [PATCH 7/9] mm: add a flag to indicate we used a cached page Date: Wed, 26 Sep 2018 17:08:54 -0400 Message-Id: <20180926210856.7895-8-josef@toxicpanda.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180926210856.7895-1-josef@toxicpanda.com> References: <20180926210856.7895-1-josef@toxicpanda.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This is preparation for dropping the mmap_sem in page_mkwrite. We need to know if we used our cached page so we can be sure it is the page we already did the page_mkwrite stuff on so we don't have to redo all of that work. Signed-off-by: Josef Bacik --- include/linux/mm.h | 6 +++++- mm/filemap.c | 5 ++++- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 4a84ec976dfc..a7305d193c71 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -318,6 +318,9 @@ extern pgprot_t protection_map[16]; #define FAULT_FLAG_USER 0x40 /* The fault originated in userspace */ #define FAULT_FLAG_REMOTE 0x80 /* faulting for non current tsk/mm */ #define FAULT_FLAG_INSTRUCTION 0x100 /* The fault was during an instruction fetch */ +#define FAULT_FLAG_USED_CACHED 0x200 /* Our vmf->page was from a previous + * loop through the fault handler. + */ #define FAULT_FLAG_TRACE \ { FAULT_FLAG_WRITE, "WRITE" }, \ @@ -328,7 +331,8 @@ extern pgprot_t protection_map[16]; { FAULT_FLAG_TRIED, "TRIED" }, \ { FAULT_FLAG_USER, "USER" }, \ { FAULT_FLAG_REMOTE, "REMOTE" }, \ - { FAULT_FLAG_INSTRUCTION, "INSTRUCTION" } + { FAULT_FLAG_INSTRUCTION, "INSTRUCTION" }, \ + { FAULT_FLAG_USED_CACHED, "USED_CACHED" } /* * vm_fault is filled by the the pagefault handler and passed to the vma's diff --git a/mm/filemap.c b/mm/filemap.c index 5212ab637832..e9cb44bd35aa 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2556,6 +2556,7 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) if (cached_page->mapping == mapping && cached_page->index == offset) { page = cached_page; + vmf->flags |= FAULT_FLAG_USED_CACHED; goto have_cached_page; } unlock_page(cached_page); @@ -2619,8 +2620,10 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) * We have a locked page in the page cache, now we need to check * that it's up-to-date. If not, it is going to be due to an error. */ - if (unlikely(!PageUptodate(page))) + if (unlikely(!PageUptodate(page))) { + vmf->flags &= ~(FAULT_FLAG_USED_CACHED); goto page_not_uptodate; + } /* * Found the page and have a reference on it. From patchwork Wed Sep 26 21:08:55 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 10616865 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 322BB15A6 for ; Wed, 26 Sep 2018 21:09:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 21B312B7F2 for ; Wed, 26 Sep 2018 21:09:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 152A22B7F9; Wed, 26 Sep 2018 21:09:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A7B952B7F2 for ; Wed, 26 Sep 2018 21:09:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DF48C8E000A; Wed, 26 Sep 2018 17:09:14 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D7EE68E0008; Wed, 26 Sep 2018 17:09:14 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C1F3B8E000A; Wed, 26 Sep 2018 17:09:14 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by kanga.kvack.org (Postfix) with ESMTP id 8CCF88E0008 for ; Wed, 26 Sep 2018 17:09:14 -0400 (EDT) Received: by mail-qk1-f198.google.com with SMTP id d205-v6so434876qkg.16 for ; Wed, 26 Sep 2018 14:09:14 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:subject:date:message-id :in-reply-to:references; bh=uNw1SEfuNh5muyIlKMjO8zXN2XjQ0Df1a1ZQWi9S3hE=; b=YfMC5r8olRMqAPosYAjaLVFMAeVR4fu8XwU4TvsBBzjENYEaS0yTuq+TXiMh1ZIpKq NRG1WXWiJxXDSKLjO744WZFvgzMT6q8kODLp87J2ZHaqFebt/rWThS+xO7s0Ck/cYuH9 RGewJQnYon3OfG7zLdUqpKkcQoQMw9tvvyIqOfBadXfQzdRKOU3XFeOyhY1uVQs74mU/ JDY4XdiXQfrPO1u6gqVo97UND8Cf2UijaTSL+T7tJloUe6VfoB1vc1m3K4EJyQVspNYe y85pypeU42mKFyootk5x7B63FfTN5fhkSlXc5/FMuybyPKodA6ZboZPestkJr4AYFJ8a AmaA== X-Gm-Message-State: ABuFfoi1GNOOsyEOGq6+BCAyCtPDufUARHVbi1ymGyigQmG+1DvCJZbw vhRdtlRa3NWALyqO4gkktnNEbFH3Rz0re2cqFyGxsL52SiC9oYyj5R0hO4HPtmzcXWrphdCbBKy At3UAqU4TRHx3wq2aJH+d+EdZwPudDF6tAlhzxSOn5mzToLlO3OeMXv10HcQLbDziLW/jp0gkQl HrLDj7bqgjYYMqYNF4BSNqDnU0XKxVDEveESvNmmxF3EaGCeDfm6ueH5f11dEDhA5aecWURQ5sA 0n/v4da7+Z/waNXZTYD8X+g9dRdY+BqFn54NoLuxptxNAt726ZbJ9hHU4P70oZH0iUbc5RWhC9b PwuWa3cadgiMNtHGAssRCu5finay1ik4N74otHqCZ1qdpSo5aReyq4Uv8fmAJjeejRQ63hNgGDB 6 X-Received: by 2002:aed:3fce:: with SMTP id w14-v6mr6001962qth.226.1537996154371; Wed, 26 Sep 2018 14:09:14 -0700 (PDT) X-Received: by 2002:aed:3fce:: with SMTP id w14-v6mr6001934qth.226.1537996153864; Wed, 26 Sep 2018 14:09:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537996153; cv=none; d=google.com; s=arc-20160816; b=0X+SxUq827c24EOIyIK7oSx8E3ZrsWhd2407Rat+Nm42EQu7N0V3f4P/MclNaH7epo Pva3wotwTq6HhY5uZv20qI6mA1nIj9Tbb20fPDj9jH1+3547dy33gtbI6nEXIu2P0tms tmfL9OF+hem2O45fJ1JXG4TJh3221FxEhl292SO1FEJuLle+dYrVzUe56phYKsipNgaZ eOZ46SU2hSBj331hVTE2AI7bEiWZ2kbOwhPxmqvVR4VGWvziZq1hMAmWi9Ab2U8U1GT2 uD0PmS3uSa/mYN08rbrarBE2rtcdpq9z6fN9um9pAug9p4kRSPyxbPI3GSP0/V5b03g4 gCGQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=uNw1SEfuNh5muyIlKMjO8zXN2XjQ0Df1a1ZQWi9S3hE=; b=yEG8qgOlbtIxsByp+JQgyfRkhX0IP4CnWTmMcgXcDDXXbB5KHgAymq/VhtPprp3pwj UCMlh/0fzdEWqD4oF7S3EOjzdhkezGzT++7nsyV0Ez0gdQAzCrkU+gx/1zMa/C0G3cPQ XdjyxGn2dIXAqCl0Ravn8e1TOQdca4RVr3PomZvQkJgahxH9ofhRam2ObE0bglM+zZyJ c62zKJfiWPIJpNODdSxsS2BhQ8VIaFm3GHiRNCP30x58Lifg6+1no3+ZK2mJ6g9hDVFv xXFgmCCaE9Jg6RTiAxvKgZFO5L93Umjz3q7Saws7dQOy9wtUfZJHn+4Z7hO27OS6EZgN tPbQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@toxicpanda-com.20150623.gappssmtp.com header.s=20150623 header.b=E+IkZtxW; spf=neutral (google.com: 209.85.220.65 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) smtp.mailfrom=josef@toxicpanda.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id b54-v6sor54187qtk.74.2018.09.26.14.09.13 for (Google Transport Security); Wed, 26 Sep 2018 14:09:13 -0700 (PDT) Received-SPF: neutral (google.com: 209.85.220.65 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@toxicpanda-com.20150623.gappssmtp.com header.s=20150623 header.b=E+IkZtxW; spf=neutral (google.com: 209.85.220.65 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) smtp.mailfrom=josef@toxicpanda.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references; bh=uNw1SEfuNh5muyIlKMjO8zXN2XjQ0Df1a1ZQWi9S3hE=; b=E+IkZtxWnldcnfnVAFchZ1hWE504qBGKdX2eijVXxr7G+Yns/Lw/m0dK3zwPRS5BR7 NPhHdsbYsqv7Rk+GL8cmQXek2L9S55HKsJVPA0JzSGx8F5+cBMI0QQ9EHQtfNLwA7i4t sC3pLwKemV+y20kKGmmL7K6hBp6n0IfS+xlPMNJ1jxbDFLwniUzBbzgyaLZ1K5mkWh7P eg9pTRdVcxR0Iz8T8p0VWPa/8okeH8kK+cdMZva7oNTlzpExw4ukGzcJj8CVcUfue3GZ UYlI0UuSGIZmFrnbVaBQ6nzmTHAXCMDv3kggv8aYdCPINTF/+3uSSqUkA/HIfFou9+kA XliA== X-Google-Smtp-Source: ACcGV62nU0p/SpQCuwEhccw6FtSj8cKf4zvclsHPNrLrKJb3VdwuIBbYKgcnnrv2S2qANr8nkWYi+g== X-Received: by 2002:ac8:39a1:: with SMTP id v30-v6mr5494082qte.58.1537996153608; Wed, 26 Sep 2018 14:09:13 -0700 (PDT) Received: from localhost ([107.15.81.208]) by smtp.gmail.com with ESMTPSA id r9-v6sm37064qta.87.2018.09.26.14.09.12 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 26 Sep 2018 14:09:12 -0700 (PDT) From: Josef Bacik To: kernel-team@fb.com, linux-kernel@vger.kernel.org, hannes@cmpxchg.org, tj@kernel.org, linux-fsdevel@vger.kernel.org, akpm@linux-foundation.org, riel@redhat.com, linux-mm@kvack.org, linux-btrfs@vger.kernel.org Subject: [PATCH 8/9] mm: allow ->page_mkwrite to do retries Date: Wed, 26 Sep 2018 17:08:55 -0400 Message-Id: <20180926210856.7895-9-josef@toxicpanda.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180926210856.7895-1-josef@toxicpanda.com> References: <20180926210856.7895-1-josef@toxicpanda.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Before we didn't set the retry flag on our vm_fault. We want to allow file systems to drop the mmap_sem if they so choose, so set this flag and deal with VM_FAULT_RETRY appropriately. Signed-off-by: Josef Bacik --- mm/memory.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 6f8abde84986..821435855177 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2384,11 +2384,13 @@ static vm_fault_t do_page_mkwrite(struct vm_fault *vmf) unsigned int old_flags = vmf->flags; vmf->flags = FAULT_FLAG_WRITE|FAULT_FLAG_MKWRITE; + vmf->flags |= old_flags & FAULT_FLAG_ALLOW_RETRY; ret = vmf->vma->vm_ops->page_mkwrite(vmf); /* Restore original flags so that caller is not surprised */ vmf->flags = old_flags; - if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE))) + if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | + VM_FAULT_RETRY))) return ret; if (unlikely(!(ret & VM_FAULT_LOCKED))) { lock_page(page); @@ -2683,7 +2685,8 @@ static vm_fault_t wp_page_shared(struct vm_fault *vmf) pte_unmap_unlock(vmf->pte, vmf->ptl); tmp = do_page_mkwrite(vmf); if (unlikely(!tmp || (tmp & - (VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) { + (VM_FAULT_ERROR | VM_FAULT_NOPAGE | + VM_FAULT_RETRY)))) { put_page(vmf->page); return tmp; } @@ -3716,7 +3719,8 @@ static vm_fault_t do_shared_fault(struct vm_fault *vmf) unlock_page(vmf->page); tmp = do_page_mkwrite(vmf); if (unlikely(!tmp || - (tmp & (VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) { + (tmp & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | + VM_FAULT_RETRY)))) { put_page(vmf->page); return tmp; } From patchwork Wed Sep 26 21:08:56 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 10616867 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2C28015A6 for ; Wed, 26 Sep 2018 21:09:34 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1B1F72B7F2 for ; Wed, 26 Sep 2018 21:09:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0DB762B7F9; Wed, 26 Sep 2018 21:09:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 733552B7F2 for ; Wed, 26 Sep 2018 21:09:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F0D7E8E000B; Wed, 26 Sep 2018 17:09:16 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id EBB1F8E0008; Wed, 26 Sep 2018 17:09:16 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D3B4D8E000B; Wed, 26 Sep 2018 17:09:16 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by kanga.kvack.org (Postfix) with ESMTP id 9B07B8E0008 for ; Wed, 26 Sep 2018 17:09:16 -0400 (EDT) Received: by mail-qt1-f200.google.com with SMTP id d18-v6so346534qtj.20 for ; Wed, 26 Sep 2018 14:09:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:subject:date:message-id :in-reply-to:references; bh=auspTN6sS8IxHotb0wwOzsGrFw8EJq3KjZuMbDyvHfw=; b=GsLjZMnb+jQ95ZRGVjBvqSCcU5xjzpyxtIEKkY5ISi5uv/6CL42Z6Ga04/ZiXxl3Dg aEbbzepD/Mc008AvQisERvNYWwqxy6cK1gCBRnP8bE+Da/RIMJpaODu2h/jzrRq1K57M 8muObe3/ooBE/zCuzHdKT6dzl/G6jyuDvkOAMCVlvJcHNuzRPsICl3NfpT79Uqburif1 NKLEZLIUuOXbqeohOOx/WRSgy8DNnirl3NOKr53fCMf+CzhxjqHpBWwmMwWINqrbL7ar K/YjTJG8ZJxMN/bR/au9wlyOh9TxdSn26lNw+qmYVRlL68pKOWBdk3evbdsQU8rfXoUK mnEA== X-Gm-Message-State: ABuFfoid5Of6dgtNN2qWQ0D+h0yGPGPrYJ9eV0EhujUSCKALbrZt0umh l7aldJuXYfX4E3kM96Q9FZSkWNSSQijNqK9ZqOhrIsVEhNEDz0GCSmnljOm9a/cJvrtUYG/yfIw /nRsc5Ms7wES8ArCu2QHQp9FHKcR7N9hzcjN6hmjyT1vPKpB/GyilxftFITYILI1z3KqXzEBAGA QRusVgWzhbpZBWpW2EZwt5l4k/J7Td6u6aigPQV7WHG6zTEqIewbWrNDVA550saB9/AKVp4Zc6j G+fWjTpUq7Tq280uHY24Ed7HiQDLemkhh+DScRWBUmPrkTQxweJdgYHFkqJchFQ+qNgVx5O6eoY L67L/7JEFBwpbDfapGYP62uBy13nhGp2clNvyMqxWlQDyTepee/2pNfXvUA2APtR4/OsAGW7+Tq v X-Received: by 2002:ac8:4649:: with SMTP id f9-v6mr6169565qto.34.1537996156388; Wed, 26 Sep 2018 14:09:16 -0700 (PDT) X-Received: by 2002:ac8:4649:: with SMTP id f9-v6mr6169528qto.34.1537996155646; Wed, 26 Sep 2018 14:09:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537996155; cv=none; d=google.com; s=arc-20160816; b=dJxThA59lVVxWK0HRzDTNYDHd4oRrqzWOpQhyYYmNqQkqAymkbiIhQMiX2Ej1ortSp ZeSUzqoginza7MTbqCjUr1RQC33wTOoIQRqOzWI2lssFTJCn8tJh2/j2AS3yIgzynn4K FJj3UYQRIuaGyAzD51RK0sqzYgwhN7RSEHScP8UInBdWT7gQmuCRjK07fVXMNtLv1AdR QLUK78JlNfmV3yHegJnQ3QElD+z4A0zIBh+N+3CyEdoPR3lmHAVJXoQhRU9Is4v7TJw2 dO7aQnAx9tufZdgD5xdiQmpWCLjy3RiEP4VSb5h7baB+Qj8jTz2Q3Pi37zeRuXD2w7Wk 2VTA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=auspTN6sS8IxHotb0wwOzsGrFw8EJq3KjZuMbDyvHfw=; b=Mc/FBLL5InGJzsnTiiYuH34AAMvI8yE2jSt/7aj90DpJSAPpQnHjhH4UYyPP68FC3I EO+k7M35MPtrKFIZJ3dLae9raZWgLYc6D6piDlcZPG3CsD/fmaEeJo0iz23sPsB+ZxAB WkuWir3NhtQ6nsG8RafAkin5e/vv0EEFbIDCnEIQ32Y8K/JkdZk1fIz5G+Wxrs1S/6L1 7/2ktC8jAo6OLiFPXYFUdmcjcmRLtnRRq4pEwWH7l6prvLMp63R3Zy7fU0mv9FKJizCE gV5WrLcdaUShWEiKpetqhxuY5d1gUSkdrGPq6qPw1KKC56gdu//JsJMyJMgFE/tm6i/R 2uMA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@toxicpanda-com.20150623.gappssmtp.com header.s=20150623 header.b=SXAd6Vke; spf=neutral (google.com: 209.85.220.65 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) smtp.mailfrom=josef@toxicpanda.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id z38-v6sor65350qtb.1.2018.09.26.14.09.15 for (Google Transport Security); Wed, 26 Sep 2018 14:09:15 -0700 (PDT) Received-SPF: neutral (google.com: 209.85.220.65 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@toxicpanda-com.20150623.gappssmtp.com header.s=20150623 header.b=SXAd6Vke; spf=neutral (google.com: 209.85.220.65 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) smtp.mailfrom=josef@toxicpanda.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references; bh=auspTN6sS8IxHotb0wwOzsGrFw8EJq3KjZuMbDyvHfw=; b=SXAd6Vke7LmvdOzgepTuMkxa5FVG4FDrn4gPRLVbWKiMjOC6JruSWoiGNuqVBPi55i oomIuyIQpOYd5drvE4MP5YkIE/T8CccT7b4a0rCb/jOWcNs/VLzth1hvdpAJIC+Kr8sS Kdql0JzF4IvjxHmTULK+GNHNHU2T1sELYyiUO3fwvg8cB+vzlHY1aCyOkAGeoo7ZuI0K q4tEdtkustrtW0fLoHEhksp0WWm6UWorJUfYwumQBeNgr0t397zLmHZW/2TKCgQPhqn4 LUcY8NcfEvfqRYiCWcGN2kwaHc9TZ9ZIWFZYX+ykxDz+aHy+hpQ5PyNpPQCYmLLRi1PJ GzFw== X-Google-Smtp-Source: ACcGV62nJC3T0WRvUixrTKzlWB2aS2+o+U13U55Ewpn8iBVNXBNucv90lx3SdSa1/k4WkjXmvdPwbA== X-Received: by 2002:aed:2b43:: with SMTP id p61-v6mr6152561qtd.107.1537996155364; Wed, 26 Sep 2018 14:09:15 -0700 (PDT) Received: from localhost ([107.15.81.208]) by smtp.gmail.com with ESMTPSA id q26-v6sm86927qta.18.2018.09.26.14.09.14 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 26 Sep 2018 14:09:14 -0700 (PDT) From: Josef Bacik To: kernel-team@fb.com, linux-kernel@vger.kernel.org, hannes@cmpxchg.org, tj@kernel.org, linux-fsdevel@vger.kernel.org, akpm@linux-foundation.org, riel@redhat.com, linux-mm@kvack.org, linux-btrfs@vger.kernel.org Subject: [PATCH 9/9] btrfs: drop mmap_sem in mkwrite for btrfs Date: Wed, 26 Sep 2018 17:08:56 -0400 Message-Id: <20180926210856.7895-10-josef@toxicpanda.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180926210856.7895-1-josef@toxicpanda.com> References: <20180926210856.7895-1-josef@toxicpanda.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP ->page_mkwrite is extremely expensive in btrfs. We have to reserve space, which can take 6 lifetimes, and we could possibly have to wait on writeback on the page, another several lifetimes. To avoid this simply drop the mmap_sem if we didn't have the cached page and do all of our work and return the appropriate retry error. If we have the cached page we know we did all the right things to set this page up and we can just carry on. Signed-off-by: Josef Bacik --- fs/btrfs/inode.c | 41 +++++++++++++++++++++++++++++++++++++++-- include/linux/mm.h | 14 ++++++++++++++ mm/filemap.c | 3 ++- 3 files changed, 55 insertions(+), 3 deletions(-) diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index 3ea5339603cf..6b723d29bc0c 100644 --- a/fs/btrfs/inode.c +++ b/fs/btrfs/inode.c @@ -8809,7 +8809,9 @@ static void btrfs_invalidatepage(struct page *page, unsigned int offset, vm_fault_t btrfs_page_mkwrite(struct vm_fault *vmf) { struct page *page = vmf->page; - struct inode *inode = file_inode(vmf->vma->vm_file); + struct file *file = vmf->vma->vm_file, *fpin; + struct mm_struct *mm = vmf->vma->vm_mm; + struct inode *inode = file_inode(file); struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb); struct extent_io_tree *io_tree = &BTRFS_I(inode)->io_tree; struct btrfs_ordered_extent *ordered; @@ -8828,6 +8830,29 @@ vm_fault_t btrfs_page_mkwrite(struct vm_fault *vmf) reserved_space = PAGE_SIZE; + /* + * We have our cached page from a previous mkwrite, check it to make + * sure it's still dirty and our file size matches when we ran mkwrite + * the last time. If everything is OK then return VM_FAULT_LOCKED, + * otherwise do the mkwrite again. + */ + if (vmf->flags & FAULT_FLAG_USED_CACHED) { + lock_page(page); + if (vmf->cached_size == i_size_read(inode) && + PageDirty(page)) + return VM_FAULT_LOCKED; + unlock_page(page); + } + + /* + * mkwrite is extremely expensive, and we are holding the mmap_sem + * during this, which means we can starve out anybody trying to + * down_write(mmap_sem) for a long while, especially if we throw cgroups + * into the mix. So just drop the mmap_sem and do all of our work, + * we'll loop back through and verify everything is ok the next time and + * hopefully avoid doing the work twice. + */ + fpin = maybe_unlock_mmap_for_io(vmf->vma, vmf->flags); sb_start_pagefault(inode->i_sb); page_start = page_offset(page); page_end = page_start + PAGE_SIZE - 1; @@ -8844,7 +8869,7 @@ vm_fault_t btrfs_page_mkwrite(struct vm_fault *vmf) ret2 = btrfs_delalloc_reserve_space(inode, &data_reserved, page_start, reserved_space); if (!ret2) { - ret2 = file_update_time(vmf->vma->vm_file); + ret2 = file_update_time(file); reserved = 1; } if (ret2) { @@ -8943,6 +8968,14 @@ vm_fault_t btrfs_page_mkwrite(struct vm_fault *vmf) btrfs_delalloc_release_extents(BTRFS_I(inode), PAGE_SIZE, true); sb_end_pagefault(inode->i_sb); extent_changeset_free(data_reserved); + if (fpin) { + unlock_page(page); + fput(fpin); + get_page(page); + vmf->cached_size = size; + vmf->cached_page = page; + return VM_FAULT_RETRY; + } return VM_FAULT_LOCKED; } @@ -8955,6 +8988,10 @@ vm_fault_t btrfs_page_mkwrite(struct vm_fault *vmf) out_noreserve: sb_end_pagefault(inode->i_sb); extent_changeset_free(data_reserved); + if (fpin) { + fput(fpin); + down_read(&mm->mmap_sem); + } return ret; } diff --git a/include/linux/mm.h b/include/linux/mm.h index a7305d193c71..9409845d0411 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -370,6 +370,13 @@ struct vm_fault { * next time we loop through the fault * handler for faster lookup. */ + loff_t cached_size; /* ->page_mkwrite handlers may drop + * the mmap_sem to avoid starvation, in + * which case they need to save the + * i_size in order to verify the cached + * page we're using the next loop + * through hasn't changed under us. + */ /* These three entries are valid only while holding ptl lock */ pte_t *pte; /* Pointer to pte entry matching * the 'address'. NULL if the page @@ -1437,6 +1444,8 @@ extern vm_fault_t handle_mm_fault_cacheable(struct vm_fault *vmf); extern int fixup_user_fault(struct task_struct *tsk, struct mm_struct *mm, unsigned long address, unsigned int fault_flags, bool *unlocked); +extern struct file *maybe_unlock_mmap_for_io(struct vm_area_struct *vma, + int flags); void unmap_mapping_pages(struct address_space *mapping, pgoff_t start, pgoff_t nr, bool even_cows); void unmap_mapping_range(struct address_space *mapping, @@ -1463,6 +1472,11 @@ static inline int fixup_user_fault(struct task_struct *tsk, BUG(); return -EFAULT; } +stiatc inline struct file *maybe_unlock_mmap_for_io(struct vm_area_struct *vma, + int flags) +{ + return NULL; +} static inline void unmap_mapping_pages(struct address_space *mapping, pgoff_t start, pgoff_t nr, bool even_cows) { } static inline void unmap_mapping_range(struct address_space *mapping, diff --git a/mm/filemap.c b/mm/filemap.c index e9cb44bd35aa..8027f082d74f 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2366,7 +2366,7 @@ generic_file_read_iter(struct kiocb *iocb, struct iov_iter *iter) EXPORT_SYMBOL(generic_file_read_iter); #ifdef CONFIG_MMU -static struct file *maybe_unlock_mmap_for_io(struct vm_area_struct *vma, int flags) +struct file *maybe_unlock_mmap_for_io(struct vm_area_struct *vma, int flags) { if ((flags & (FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_RETRY_NOWAIT)) == FAULT_FLAG_ALLOW_RETRY) { struct file *file; @@ -2377,6 +2377,7 @@ static struct file *maybe_unlock_mmap_for_io(struct vm_area_struct *vma, int fla } return NULL; } +EXPORT_SYMBOL_GPL(maybe_unlock_mmap_for_io); /** * page_cache_read - adds requested page to the page cache if not already there