From patchwork Thu Oct 18 20:23:13 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 10648095 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B1E3C17D4 for ; Thu, 18 Oct 2018 20:23:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A607128D91 for ; Thu, 18 Oct 2018 20:23:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9A3AF28D72; Thu, 18 Oct 2018 20:23:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C76EF28D8C for ; Thu, 18 Oct 2018 20:23:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C82DC6B0008; Thu, 18 Oct 2018 16:23:32 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C09426B000C; Thu, 18 Oct 2018 16:23:32 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A87986B000D; Thu, 18 Oct 2018 16:23:32 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by kanga.kvack.org (Postfix) with ESMTP id 78C5E6B0008 for ; Thu, 18 Oct 2018 16:23:32 -0400 (EDT) Received: by mail-qt1-f198.google.com with SMTP id x7-v6so33507447qtb.6 for ; Thu, 18 Oct 2018 13:23:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:subject:date:message-id :in-reply-to:references; bh=FY4PvfFFgkG4rCGf8NvSi/OKKAo2xpKioHzicQvhcOI=; b=qQQJIwB5yLMNmbGofEO2X/TkufDp8leV+edWFGed2YtMtlSifwmZfiCkaCNcT7S3t2 c5QXcXSqjLinWH4h3cIqTf52k/xvjuocTDQ42TymSK49E3a3sNFRWGU3QQpr/rYc/2cH Q0TYnQr2QwQI1sfA0DQlQOIDgibuXF8X78ieUBCKhen5Dc/ezMbyjAIzEbY+1PSbvEDl HhHAXM2SIeg2c8Z0dQ1YWlz2CTpfDnz8tbIsG3Paxuq45FrgUm0/L7/plbRJ+c1wgw2w BqmxQxiECbEVMTcLLY+2YwflkZR1mDKpe/Brd1JPU8hELV5SL69v7NA+tLflnpOdwP0J XnpQ== X-Gm-Message-State: ABuFfohywbfs2+29iymRSZLgeLNLEFr+myKeIqUE1Ji2bACEhWf40xmy IhQ5lTLXIr45iidoFm8v4QT4VwNw/Y3OnbWo0EKLiIZ+974XYzgafvwrLbVL4JG49pli8XpBIJ/ t28kdjX1N5AGlhACtYAeVkkvjted84Es5owVVAMQyFxBxUJhV3Uy+YcjM7dcYsYJogfGM4lvJ2Z 3Vvf3Z+7+Lcgqf28vOKVpADHIsETQ93iAkdlrnKvyOaaIwB0XwfjQB5ACjCILS9dsYH0TyyjDZP 2EM/AajtQzTXFNoXDI/JNfs71f8A3VhCXrzPK/uCNAbDiYMkimS3dnj91rx3+DEOXobPtpPrlbv oB+yFrts9d8rY/CmA9+e8Q51BMIfxxYH0H+EodY5WlfnfxInLJkmzcVOtntApDVF7bcJZ1fNRXR N X-Received: by 2002:ac8:5548:: with SMTP id o8-v6mr28999258qtr.283.1539894212208; Thu, 18 Oct 2018 13:23:32 -0700 (PDT) X-Received: by 2002:ac8:5548:: with SMTP id o8-v6mr28999197qtr.283.1539894211249; Thu, 18 Oct 2018 13:23:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539894211; cv=none; d=google.com; s=arc-20160816; b=oNnBL36Zub0POSnZ0mMUS28BM6rBKnVIPzLeEtJrHsId0V0urPP2KU+ufrOh1TAN3m 1+pFWlPFhYY1cTQ4qAed6VRS2V4u6lj3ZuqIj2gp0Zz17gUAJr79QQ49M0gGLnXbv9UJ efXbZMmp/rP4JKUzJIE9EUKAPasZJlrwt4jlsugu9w2tfb6NS238ZUt7vLB/Wdlzvc9L ped618qzyjCslet5ebx8jrhC7Rb2Qdfidyn4zRzF76qGHjnZhksQz/piAsTiExqEMOsG WgHkGRuXFWzMiChRi/mWDbm8cKGoZrUf4OwLhec+XErr5IMiz5Z8/CZYKUrJMBT+kwPo cS5A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=FY4PvfFFgkG4rCGf8NvSi/OKKAo2xpKioHzicQvhcOI=; b=p8z9yMDHCsCKALaOmli2L6ckf1Z0pFE0JIOomgDm7hI2yBitKKMEh4AgpU+cUtfiRY iS5NbVIQbAIgO2YKvbx3xCM+DgsZUikLkj6ed/kcYokknBa3Kyx7QMirUvBxQFA9GzP3 t/em63nFwr7mz+JdE3mqi1YfTlAqJBLxcaNAXtIWPp7ISZlp3YOTgNCsYW7aAVKukmDj iQ+VGlZTWF6s9MuQw3sbo0OwiXcGrQtuqoKqb3biK9OcIagf5YwERBm/aGKSBxgSyb8d wwK4K2DKjDeZ0UgoDtm8yKlJAUJRXX9F1nfiaeUIMdcfvm/OcAvMaNkyZf07pmA47VW2 32Yw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@toxicpanda-com.20150623.gappssmtp.com header.s=20150623 header.b="L5/pCy/g"; spf=neutral (google.com: 209.85.220.65 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) smtp.mailfrom=josef@toxicpanda.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id 22-v6sor25716442qtm.73.2018.10.18.13.23.31 for (Google Transport Security); Thu, 18 Oct 2018 13:23:31 -0700 (PDT) Received-SPF: neutral (google.com: 209.85.220.65 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@toxicpanda-com.20150623.gappssmtp.com header.s=20150623 header.b="L5/pCy/g"; spf=neutral (google.com: 209.85.220.65 is neither permitted nor denied by best guess record for domain of josef@toxicpanda.com) smtp.mailfrom=josef@toxicpanda.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references; bh=FY4PvfFFgkG4rCGf8NvSi/OKKAo2xpKioHzicQvhcOI=; b=L5/pCy/gPsV0ofzn3TUULb+VIWeVJ1YdWREgQOC05qgetxeyc/gooBv0/7P1TprVEK zN3AcQkua6WC4y8j+OVVn7h38ObbMlDCmLRGDMONosHbBPAVjdTFsmicgMz49RblXBQs IcnthT0aIsVcZyB7Eb40druS3N91WGeYrki035XZGFqCdIWvsh9in2sHO3sBkEm7iiEm l7DczhjMfarG3XTGID9ITMRzBXWvJrVDQ9+Ul4n807bdgXABCoMOfrbhGGhPLAAdjBnO eoQSrbg/V/JVnhs9nsuAPDj1AfS1d/J+QoPyTfAqdqe912W2H1BZnvXfcbZ4Xy4OfptH otrw== X-Google-Smtp-Source: ACcGV60gZjzwHJ0OsBPoG+m0NpaS9G+lB3FxI11r5t1/cPnWTBZZDswZrgJkgarGYA3Zn1iwTLsU3g== X-Received: by 2002:aed:3fb9:: with SMTP id s54-v6mr29909677qth.208.1539894210875; Thu, 18 Oct 2018 13:23:30 -0700 (PDT) Received: from localhost ([107.15.81.208]) by smtp.gmail.com with ESMTPSA id m71-v6sm11781158qke.71.2018.10.18.13.23.29 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 18 Oct 2018 13:23:29 -0700 (PDT) From: Josef Bacik To: kernel-team@fb.com, hannes@cmpxchg.org, linux-kernel@vger.kernel.org, tj@kernel.org, david@fromorbit.com, akpm@linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-btrfs@vger.kernel.org, riel@fb.com, linux-mm@kvack.org Subject: [PATCH 2/7] mm: drop mmap_sem for page cache read IO submission Date: Thu, 18 Oct 2018 16:23:13 -0400 Message-Id: <20181018202318.9131-3-josef@toxicpanda.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20181018202318.9131-1-josef@toxicpanda.com> References: <20181018202318.9131-1-josef@toxicpanda.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Johannes Weiner Reads can take a long time, and if anybody needs to take a write lock on the mmap_sem it'll block any subsequent readers to the mmap_sem while the read is outstanding, which could cause long delays. Instead drop the mmap_sem if we do any reads at all. Signed-off-by: Johannes Weiner Signed-off-by: Josef Bacik --- mm/filemap.c | 119 ++++++++++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 90 insertions(+), 29 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 52517f28e6f4..1ed35cd99b2c 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2366,6 +2366,18 @@ generic_file_read_iter(struct kiocb *iocb, struct iov_iter *iter) EXPORT_SYMBOL(generic_file_read_iter); #ifdef CONFIG_MMU +static struct file *maybe_unlock_mmap_for_io(struct vm_area_struct *vma, int flags) +{ + if ((flags & (FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_RETRY_NOWAIT)) == FAULT_FLAG_ALLOW_RETRY) { + struct file *file; + + file = get_file(vma->vm_file); + up_read(&vma->vm_mm->mmap_sem); + return file; + } + return NULL; +} + /** * page_cache_read - adds requested page to the page cache if not already there * @file: file to read @@ -2405,23 +2417,28 @@ static int page_cache_read(struct file *file, pgoff_t offset, gfp_t gfp_mask) * Synchronous readahead happens when we don't even find * a page in the page cache at all. */ -static void do_sync_mmap_readahead(struct vm_area_struct *vma, - struct file_ra_state *ra, - struct file *file, - pgoff_t offset) +static int do_sync_mmap_readahead(struct vm_area_struct *vma, + struct file_ra_state *ra, + struct file *file, + pgoff_t offset, + int flags) { struct address_space *mapping = file->f_mapping; + struct file *fpin; /* If we don't want any read-ahead, don't bother */ if (vma->vm_flags & VM_RAND_READ) - return; + return 0; if (!ra->ra_pages) - return; + return 0; if (vma->vm_flags & VM_SEQ_READ) { + fpin = maybe_unlock_mmap_for_io(vma, flags); page_cache_sync_readahead(mapping, ra, file, offset, ra->ra_pages); - return; + if (fpin) + fput(fpin); + return fpin ? -EAGAIN : 0; } /* Avoid banging the cache line if not needed */ @@ -2433,7 +2450,9 @@ static void do_sync_mmap_readahead(struct vm_area_struct *vma, * stop bothering with read-ahead. It will only hurt. */ if (ra->mmap_miss > MMAP_LOTSAMISS) - return; + return 0; + + fpin = maybe_unlock_mmap_for_io(vma, flags); /* * mmap read-around @@ -2442,28 +2461,40 @@ static void do_sync_mmap_readahead(struct vm_area_struct *vma, ra->size = ra->ra_pages; ra->async_size = ra->ra_pages / 4; ra_submit(ra, mapping, file); + + if (fpin) + fput(fpin); + + return fpin ? -EAGAIN : 0; } /* * Asynchronous readahead happens when we find the page and PG_readahead, * so we want to possibly extend the readahead further.. */ -static void do_async_mmap_readahead(struct vm_area_struct *vma, - struct file_ra_state *ra, - struct file *file, - struct page *page, - pgoff_t offset) +static int do_async_mmap_readahead(struct vm_area_struct *vma, + struct file_ra_state *ra, + struct file *file, + struct page *page, + pgoff_t offset, + int flags) { struct address_space *mapping = file->f_mapping; + struct file *fpin; /* If we don't want any read-ahead, don't bother */ if (vma->vm_flags & VM_RAND_READ) - return; + return 0; if (ra->mmap_miss > 0) ra->mmap_miss--; - if (PageReadahead(page)) - page_cache_async_readahead(mapping, ra, file, - page, offset, ra->ra_pages); + if (!PageReadahead(page)) + return 0; + fpin = maybe_unlock_mmap_for_io(vma, flags); + page_cache_async_readahead(mapping, ra, file, + page, offset, ra->ra_pages); + if (fpin) + fput(fpin); + return fpin ? -EAGAIN : 0; } /** @@ -2479,10 +2510,8 @@ static void do_async_mmap_readahead(struct vm_area_struct *vma, * * vma->vm_mm->mmap_sem must be held on entry. * - * If our return value has VM_FAULT_RETRY set, it's because - * lock_page_or_retry() returned 0. - * The mmap_sem has usually been released in this case. - * See __lock_page_or_retry() for the exception. + * If our return value has VM_FAULT_RETRY set, the mmap_sem has + * usually been released. * * If our return value does not have VM_FAULT_RETRY set, the mmap_sem * has not been released. @@ -2492,11 +2521,13 @@ static void do_async_mmap_readahead(struct vm_area_struct *vma, vm_fault_t filemap_fault(struct vm_fault *vmf) { int error; + struct mm_struct *mm = vmf->vma->vm_mm; struct file *file = vmf->vma->vm_file; struct address_space *mapping = file->f_mapping; struct file_ra_state *ra = &file->f_ra; struct inode *inode = mapping->host; pgoff_t offset = vmf->pgoff; + int flags = vmf->flags; pgoff_t max_off; struct page *page; vm_fault_t ret = 0; @@ -2509,27 +2540,44 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) * Do we have something in the page cache already? */ page = find_get_page(mapping, offset); - if (likely(page) && !(vmf->flags & FAULT_FLAG_TRIED)) { + if (likely(page) && !(flags & FAULT_FLAG_TRIED)) { /* * We found the page, so try async readahead before * waiting for the lock. */ - do_async_mmap_readahead(vmf->vma, ra, file, page, offset); + error = do_async_mmap_readahead(vmf->vma, ra, file, page, offset, vmf->flags); + if (error == -EAGAIN) + goto out_retry_wait; } else if (!page) { /* No page in the page cache at all */ - do_sync_mmap_readahead(vmf->vma, ra, file, offset); - count_vm_event(PGMAJFAULT); - count_memcg_event_mm(vmf->vma->vm_mm, PGMAJFAULT); ret = VM_FAULT_MAJOR; + count_vm_event(PGMAJFAULT); + count_memcg_event_mm(mm, PGMAJFAULT); + error = do_sync_mmap_readahead(vmf->vma, ra, file, offset, vmf->flags); + if (error == -EAGAIN) + goto out_retry_wait; retry_find: page = find_get_page(mapping, offset); if (!page) goto no_cached_page; } - if (!lock_page_or_retry(page, vmf->vma->vm_mm, vmf->flags)) { - put_page(page); - return ret | VM_FAULT_RETRY; + if (!trylock_page(page)) { + if (flags & FAULT_FLAG_ALLOW_RETRY) { + if (flags & FAULT_FLAG_RETRY_NOWAIT) + goto out_retry; + up_read(&mm->mmap_sem); + goto out_retry_wait; + } + if (flags & FAULT_FLAG_KILLABLE) { + int ret = __lock_page_killable(page); + + if (ret) { + up_read(&mm->mmap_sem); + goto out_retry; + } + } else + __lock_page(page); } /* Did it get truncated? */ @@ -2607,6 +2655,19 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) /* Things didn't work out. Return zero to tell the mm layer so. */ shrink_readahead_size_eio(file, ra); return VM_FAULT_SIGBUS; + +out_retry_wait: + if (page) { + if (flags & FAULT_FLAG_KILLABLE) + wait_on_page_locked_killable(page); + else + wait_on_page_locked(page); + } + +out_retry: + if (page) + put_page(page); + return ret | VM_FAULT_RETRY; } EXPORT_SYMBOL(filemap_fault);