From patchwork Tue May 16 13:36:27 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Trond Myklebust X-Patchwork-Id: 9729227 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id A18C060386 for ; Tue, 16 May 2017 13:36:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9425A28A24 for ; Tue, 16 May 2017 13:36:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 88FD128A26; Tue, 16 May 2017 13:36:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.3 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI, RCVD_IN_SORBS_SPAM, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F045C28A24 for ; Tue, 16 May 2017 13:36:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753033AbdEPNgh (ORCPT ); Tue, 16 May 2017 09:36:37 -0400 Received: from mail-io0-f193.google.com ([209.85.223.193]:33397 "EHLO mail-io0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753070AbdEPNgg (ORCPT ); Tue, 16 May 2017 09:36:36 -0400 Received: by mail-io0-f193.google.com with SMTP id m4so14442586ioe.0 for ; Tue, 16 May 2017 06:36:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:subject:date:message-id:in-reply-to:references; bh=QcsoddHazsWgfADsxlahqDdIfKZ71KmmroN6BeNvkxc=; b=gdlXdmdKwRIadZizC+IKLMMhpICDOJzy/28u3UXCXPdsfECrk5+3WnezI6+YhgEmCW 1TTLU6r3D0APiYxbRKAY8fm57Og+EZTpJQ+dr3wzqg6QoyOMR/RLKmh4WnVxF3ljGvDA 7m+ocERFX6n73QgoUiirrn9wJGyX2pPmBYNfPmG5TKftlPam1eVJjd2q0L1pyomVs6P5 OfLHfbSst6MjPUTI+gYKVNIEoWyjEAK4rlDUBExGDIrr/mCRlFD4KtpQXX6n2lBzHMSb UhWcmopCBhogfvYfoiaSp3w1rPUgUAMZQDHdZKDPom/ozAio8z63tp1F5ULCRhOWqw0B xitA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:subject:date:message-id :in-reply-to:references; bh=QcsoddHazsWgfADsxlahqDdIfKZ71KmmroN6BeNvkxc=; b=HLB/b+FtApOyix8MIe6sK3tQuaybtlvO2QN54HwRQ+EK6M4PyIEvJ+fnMyiYfTjFJH TmilnUEAAX0vrVe/eVTfcTCOi5sSt5tyQP17ZMUtUWnSmXohhPus/KE2Nj3imH4Ybn8Z SUFC4avORz/+rOdknEv2f9YKP1pNM1WLNH4irAwFhOgLL0xrpf4FT0+1JyLnHcblXrzs P3VNdKcaXgG7jCRLRtunfnUtzRdkkJoAa0bKylb6Db6hQgU6bqHvLRDjiI/NCK8GIvjs qb4SlU3afUhrmrY+fO+0PzEr+seiwsgxw8Ld1fbQim2cfV7x2f4o2ooNhfZwd9W0Hvxu eiHQ== X-Gm-Message-State: AODbwcDeZyWiufNswsb1tEmbpc8pOw3UrRen7xL1/9i+IIP5RJ8YyFRd ouvCZaON/qfsSQ== X-Received: by 10.107.181.68 with SMTP id e65mr12551520iof.156.1494941796044; Tue, 16 May 2017 06:36:36 -0700 (PDT) Received: from localhost.localdomain (c-68-49-162-121.hsd1.mi.comcast.net. [68.49.162.121]) by smtp.gmail.com with ESMTPSA id t184sm2245436iod.29.2017.05.16.06.36.35 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 May 2017 06:36:35 -0700 (PDT) From: Trond Myklebust To: linux-nfs@vger.kernel.org Subject: [PATCH 2/3] NFS: Ensure we commit after writeback is complete Date: Tue, 16 May 2017 09:36:27 -0400 Message-Id: <20170516133628.16525-3-trond.myklebust@primarydata.com> X-Mailer: git-send-email 2.9.4 In-Reply-To: <20170516133628.16525-2-trond.myklebust@primarydata.com> References: <20170516133628.16525-1-trond.myklebust@primarydata.com> <20170516133628.16525-2-trond.myklebust@primarydata.com> Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP If the page cache is being flushed, then we want to ensure that we do start a commit once the pages are done being flushed. If we just wait until all I/O is done to that file, we can end up livelocking until the balance_dirty_pages() mechanism puts its foot down and forces I/O to stop. So instead we do more or less the same thing that O_DIRECT does, and set up a counter to tell us when the flush is done, Signed-off-by: Trond Myklebust --- fs/nfs/pagelist.c | 3 +++ fs/nfs/write.c | 57 ++++++++++++++++++++++++++++++++++++++++++++++++ include/linux/nfs_page.h | 1 + include/linux/nfs_xdr.h | 2 ++ 4 files changed, 63 insertions(+) diff --git a/fs/nfs/pagelist.c b/fs/nfs/pagelist.c index de107d866297..83d2918f1b13 100644 --- a/fs/nfs/pagelist.c +++ b/fs/nfs/pagelist.c @@ -50,6 +50,7 @@ void nfs_pgheader_init(struct nfs_pageio_descriptor *desc, hdr->cred = hdr->req->wb_context->cred; hdr->io_start = req_offset(hdr->req); hdr->good_bytes = mirror->pg_count; + hdr->io_completion = desc->pg_io_completion; hdr->dreq = desc->pg_dreq; hdr->release = release; hdr->completion_ops = desc->pg_completion_ops; @@ -709,6 +710,7 @@ void nfs_pageio_init(struct nfs_pageio_descriptor *desc, desc->pg_ioflags = io_flags; desc->pg_error = 0; desc->pg_lseg = NULL; + desc->pg_io_completion = NULL; desc->pg_dreq = NULL; desc->pg_bsize = bsize; @@ -1231,6 +1233,7 @@ int nfs_pageio_resend(struct nfs_pageio_descriptor *desc, { LIST_HEAD(failed); + desc->pg_io_completion = hdr->io_completion; desc->pg_dreq = hdr->dreq; while (!list_empty(&hdr->pages)) { struct nfs_page *req = nfs_list_entry(hdr->pages.next); diff --git a/fs/nfs/write.c b/fs/nfs/write.c index db7ba542559e..051197cb9195 100644 --- a/fs/nfs/write.c +++ b/fs/nfs/write.c @@ -40,6 +40,12 @@ #define MIN_POOL_WRITE (32) #define MIN_POOL_COMMIT (4) +struct nfs_io_completion { + void (*complete)(void *data); + void *data; + struct kref refcount; +}; + /* * Local function declarations */ @@ -108,6 +114,39 @@ static void nfs_writehdr_free(struct nfs_pgio_header *hdr) mempool_free(hdr, nfs_wdata_mempool); } +static struct nfs_io_completion *nfs_io_completion_alloc(gfp_t gfp_flags) +{ + return kmalloc(sizeof(struct nfs_io_completion), gfp_flags); +} + +static void nfs_io_completion_init(struct nfs_io_completion *ioc, + void (*complete)(void *), void *data) +{ + ioc->complete = complete; + ioc->data = data; + kref_init(&ioc->refcount); +} + +static void nfs_io_completion_release(struct kref *kref) +{ + struct nfs_io_completion *ioc = container_of(kref, + struct nfs_io_completion, refcount); + ioc->complete(ioc->data); + kfree(ioc); +} + +static void nfs_io_completion_get(struct nfs_io_completion *ioc) +{ + if (ioc != NULL) + kref_get(&ioc->refcount); +} + +static void nfs_io_completion_put(struct nfs_io_completion *ioc) +{ + if (ioc != NULL) + kref_put(&ioc->refcount, nfs_io_completion_release); +} + static void nfs_context_set_write_error(struct nfs_open_context *ctx, int error) { ctx->error = error; @@ -681,18 +720,29 @@ static int nfs_writepages_callback(struct page *page, struct writeback_control * return ret; } +static void nfs_io_completion_commit(void *inode) +{ + nfs_commit_inode(inode, 0); +} + int nfs_writepages(struct address_space *mapping, struct writeback_control *wbc) { struct inode *inode = mapping->host; struct nfs_pageio_descriptor pgio; + struct nfs_io_completion *ioc = nfs_io_completion_alloc(GFP_NOFS); int err; nfs_inc_stats(inode, NFSIOS_VFSWRITEPAGES); + if (ioc) + nfs_io_completion_init(ioc, nfs_io_completion_commit, inode); + nfs_pageio_init_write(&pgio, inode, wb_priority(wbc), false, &nfs_async_write_completion_ops); + pgio.pg_io_completion = ioc; err = write_cache_pages(mapping, wbc, nfs_writepages_callback, &pgio); nfs_pageio_complete(&pgio); + nfs_io_completion_put(ioc); if (err < 0) goto out_err; @@ -940,6 +990,11 @@ int nfs_write_need_commit(struct nfs_pgio_header *hdr) return hdr->verf.committed != NFS_FILE_SYNC; } +static void nfs_async_write_init(struct nfs_pgio_header *hdr) +{ + nfs_io_completion_get(hdr->io_completion); +} + static void nfs_write_completion(struct nfs_pgio_header *hdr) { struct nfs_commit_info cinfo; @@ -973,6 +1028,7 @@ static void nfs_write_completion(struct nfs_pgio_header *hdr) nfs_release_request(req); } out: + nfs_io_completion_put(hdr->io_completion); hdr->release(hdr); } @@ -1378,6 +1434,7 @@ static void nfs_async_write_reschedule_io(struct nfs_pgio_header *hdr) } static const struct nfs_pgio_completion_ops nfs_async_write_completion_ops = { + .init_hdr = nfs_async_write_init, .error_cleanup = nfs_async_write_error, .completion = nfs_write_completion, .reschedule_io = nfs_async_write_reschedule_io, diff --git a/include/linux/nfs_page.h b/include/linux/nfs_page.h index 6138cf91346b..abbee2d15dce 100644 --- a/include/linux/nfs_page.h +++ b/include/linux/nfs_page.h @@ -93,6 +93,7 @@ struct nfs_pageio_descriptor { const struct rpc_call_ops *pg_rpc_callops; const struct nfs_pgio_completion_ops *pg_completion_ops; struct pnfs_layout_segment *pg_lseg; + struct nfs_io_completion *pg_io_completion; struct nfs_direct_req *pg_dreq; unsigned int pg_bsize; /* default bsize for mirrors */ diff --git a/include/linux/nfs_xdr.h b/include/linux/nfs_xdr.h index ee998ed08cf6..ab6a8381459f 100644 --- a/include/linux/nfs_xdr.h +++ b/include/linux/nfs_xdr.h @@ -1422,6 +1422,7 @@ enum { NFS_IOHDR_STAT, }; +struct nfs_io_completion; struct nfs_pgio_header { struct inode *inode; struct rpc_cred *cred; @@ -1435,6 +1436,7 @@ struct nfs_pgio_header { void (*release) (struct nfs_pgio_header *hdr); const struct nfs_pgio_completion_ops *completion_ops; const struct nfs_rw_ops *rw_ops; + struct nfs_io_completion *io_completion; struct nfs_direct_req *dreq; spinlock_t lock; /* fields protected by lock */