From patchwork Tue Aug 26 14:10:13 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boaz Harrosh X-Patchwork-Id: 4782001 Return-Path: X-Original-To: patchwork-linux-nfs@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork2.web.kernel.org (Postfix) with ESMTP id AA7BCC0338 for ; Tue, 26 Aug 2014 14:10:34 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 00E8020154 for ; Tue, 26 Aug 2014 14:10:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id F17FD20127 for ; Tue, 26 Aug 2014 14:10:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755254AbaHZOKT (ORCPT ); Tue, 26 Aug 2014 10:10:19 -0400 Received: from mail-wi0-f174.google.com ([209.85.212.174]:55930 "EHLO mail-wi0-f174.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755234AbaHZOKR (ORCPT ); Tue, 26 Aug 2014 10:10:17 -0400 Received: by mail-wi0-f174.google.com with SMTP id d1so4278974wiv.1 for ; Tue, 26 Aug 2014 07:10:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=VhlBcSiLbp5Gnm77oo368lFe7PS3Su44TUmbzUeGKJA=; b=mAi6A4h0j+qaoI4HdXy47uTeA2iKYVk9o2A0pRZCs8CAqZqIdiukmdRqqrXH2MC+IB tllGFGHptPEW5v1lghPAHWJHdomqfXOJLW+IdGANb6T6XwcHukLbF8yVjM9PcHO9v1oJ MLS8AHTYw0A5ho9og2srlaEmoez1a3qa64DcvURRsuogjfem+t0xJ4NNupDtw6/lKuEK 3staLVHMB9/NhGu+44yVqag0p4avfoON2jtSBTKKVqEJSsKohk3xVy8Lv5PxzBKTb3QP ydQnoaGHsQ+YTCGUmE76TPSCzpvdw90M4LySLU3+g+9ayhpW8Z8rxQZ2SIKxckQ+bTvp Kfkw== X-Gm-Message-State: ALoCoQleKbayOGWj2PLbo1qipUZqeoJEJaesaKm78xUUBzTA7/9KW8KlGcpbeb8VAEjgNpuAwRSD X-Received: by 10.180.101.162 with SMTP id fh2mr21666367wib.11.1409062215843; Tue, 26 Aug 2014 07:10:15 -0700 (PDT) Received: from [10.0.0.5] ([207.232.55.62]) by mx.google.com with ESMTPSA id lv7sm2814764wic.16.2014.08.26.07.10.14 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 26 Aug 2014 07:10:15 -0700 (PDT) Message-ID: <53FC9545.4000800@plexistor.com> Date: Tue, 26 Aug 2014 17:10:13 +0300 From: Boaz Harrosh User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.7.0 MIME-Version: 1.0 To: Christoph Hellwig CC: linux-nfs@vger.kernel.org, Matt Benjamin Subject: [PATCH] pnfs: Kick a pnfs_layoutcommit_inode on recall References: <1408637375-11343-1-git-send-email-hch@lst.de> <1408637375-11343-4-git-send-email-hch@lst.de> <53FA259C.9050807@gmail.com> <20140824191839.GA9717@lst.de> In-Reply-To: <20140824191839.GA9717@lst.de> Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Boaz Harrosh This fixes a dead-lock in the pnfs recall processing pnfs_layoutcommit_inode() is called through update_inode() called from VFS. By setting set_inode_dirty during pnfs write IO. But the VFS will not schedule another update_inode() If it is already inside an update_inode() or an sb-writeback As part of writeback pnfs code might get stuck in LAYOUT_GET with the server returning ERR_RECALL_CONFLICT because some operation has caused the server to RECALL all layouts, including those from our client. So the RECALL is received, but our client is returning ERR_DELAY because its write-segments need a LAYOUT_COMMIT, but pnfs_layoutcommit_inode will never come because it is scheduled behind the LAYOUT_GET which is stuck waiting for the recall to finish Hence the deadlock, client is stuck polling LAYOUT_GET receiving ERR_RECALL_CONFLICT. Server is stuck polling RECALL receiving ERR_DELAY. With pnfs-objects the above condition can easily happen, when a file grows beyond a group of devices. The pnfs-objects-server will RECALL all layouts because the file-objects-map will change and all old layouts will have stale attributes, therefor the RECALL is initiated as part of a LAYOUT_GET, and this can be triggered from within a single client operation. A simple solution is to kick out a pnfs_layoutcommit_inode() from within the recall, to free any need-to-commit segments and let the client return success on the RECALL, so streaming can continue. This patch Is based on 3.17-rc1. It is completely UNTESTED. I have tested a version of this patch at around the 3.12 Kernel at which point the deadlock was resolved but I hit some race conditions on pnfs state management farther on, so the actual overall processing was not fixed. But hopefully these were fixed by Trond and Christoph, and it should work better now. Signed-off-by: Boaz Harrosh Reviewed-by: Boaz Harrosh --- fs/nfs/callback_proc.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/fs/nfs/callback_proc.c b/fs/nfs/callback_proc.c index 41db525..8660f96 100644 --- a/fs/nfs/callback_proc.c +++ b/fs/nfs/callback_proc.c @@ -171,6 +171,14 @@ static u32 initiate_file_draining(struct nfs_client *clp, goto out; ino = lo->plh_inode; + + spin_lock(&ino->i_lock); + pnfs_set_layout_stateid(lo, &args->cbl_stateid, true); + spin_unlock(&ino->i_lock); + + /* kick out any segs held by need to commit */ + pnfs_layoutcommit_inode(ino, true); + spin_lock(&ino->i_lock); if (test_bit(NFS_LAYOUT_BULK_RECALL, &lo->plh_flags) || pnfs_mark_matching_lsegs_invalid(lo, &free_me_list, @@ -178,7 +186,6 @@ static u32 initiate_file_draining(struct nfs_client *clp, rv = NFS4ERR_DELAY; else rv = NFS4ERR_NOMATCHING_LAYOUT; - pnfs_set_layout_stateid(lo, &args->cbl_stateid, true); spin_unlock(&ino->i_lock); pnfs_free_lseg_list(&free_me_list); pnfs_put_layout_hdr(lo);