From patchwork Fri Apr 9 20:27:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Morton X-Patchwork-Id: 12195029 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2ED7FC43461 for ; Fri, 9 Apr 2021 20:27:32 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D28F361181 for ; Fri, 9 Apr 2021 20:27:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D28F361181 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 781696B007D; Fri, 9 Apr 2021 16:27:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 757D86B007E; Fri, 9 Apr 2021 16:27:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 61FB76B0080; Fri, 9 Apr 2021 16:27:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0237.hostedemail.com [216.40.44.237]) by kanga.kvack.org (Postfix) with ESMTP id 469B06B007D for ; Fri, 9 Apr 2021 16:27:31 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 09B208248047 for ; Fri, 9 Apr 2021 20:27:31 +0000 (UTC) X-FDA: 78013963902.03.43B01E2 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf15.hostedemail.com (Postfix) with ESMTP id 8F1ABA000146 for ; Fri, 9 Apr 2021 20:27:29 +0000 (UTC) Received: by mail.kernel.org (Postfix) with ESMTPSA id 8D88E61007; Fri, 9 Apr 2021 20:27:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1618000050; bh=YGC9Pvf473zr0zi3Ecg30zMYak7MQT5YIzwQHvlltq0=; h=Date:From:To:Subject:In-Reply-To:From; b=lGvAj5DcPP4xyMnlIbLKv3FVWLwuVemEgiDhJrUjOf/kcg7eb+3FUZGhNHUky2brh OORxem6ix+Mu3tMVGUJHArliE2rJc49pDEeWWJR2ca0hXuZA4EFca77N3N5peoKFSn Bvyx8vo4us7AwbSKU97JUEHzjr0Ep0bbpzF28zjc= Date: Fri, 09 Apr 2021 13:27:29 -0700 From: Andrew Morton To: akpm@linux-foundation.org, gechangwei@live.cn, ghe@suse.com, jlbec@evilplan.org, joseph.qi@linux.alibaba.com, junxiao.bi@oracle.com, linux-mm@kvack.org, mark@fasheh.com, mm-commits@vger.kernel.org, piaojun@huawei.com, stable@vger.kernel.org, torvalds@linux-foundation.org, wen.gang.wang@oracle.com Subject: [patch 10/16] ocfs2: fix deadlock between setattr and dio_end_io_write Message-ID: <20210409202729.4CXpdnX6s%akpm@linux-foundation.org> In-Reply-To: <20210409132633.6855fc8fea1b3905ea1bb4be@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Rspamd-Queue-Id: 8F1ABA000146 X-Stat-Signature: mtfugqyb5md9ie9yybndo6srsf6g6c1r X-Rspamd-Server: rspam02 Received-SPF: none (linux-foundation.org>: No applicable sender policy available) receiver=imf15; identity=mailfrom; envelope-from=""; helo=mail.kernel.org; client-ip=198.145.29.99 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1618000049-341041 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Wengang Wang Subject: ocfs2: fix deadlock between setattr and dio_end_io_write The following deadlock is detected: truncate -> setattr path is waiting for pending direct IO to be done ( inode->i_dio_count become zero) with inode->i_rwsem held (down_write). PID: 14827 TASK: ffff881686a9af80 CPU: 20 COMMAND: "ora_p005_hrltd9" #0 [ffffc9000bcf3c08] __schedule at ffffffff818667cc #1 [ffffc9000bcf3ca0] schedule at ffffffff81866de6 #2 [ffffc9000bcf3cb8] inode_dio_wait at ffffffff812a2d04 #3 [ffffc9000bcf3d28] ocfs2_setattr at ffffffffc05f322e [ocfs2] #4 [ffffc9000bcf3e18] notify_change at ffffffff812a5a09 #5 [ffffc9000bcf3e60] do_truncate at ffffffff812808f5 #6 [ffffc9000bcf3ed8] do_sys_ftruncate.constprop.18 at ffffffff81280cf2 #7 [ffffc9000bcf3f18] sys_ftruncate at ffffffff81280d8e #8 [ffffc9000bcf3f28] do_syscall_64 at ffffffff81003949 #9 [ffffc9000bcf3f50] entry_SYSCALL_64_after_hwframe at ffffffff81a001ad dio completion path is going to complete one direct IO (decrement inode->i_dio_count), but before that it hang at locking inode->i_rwsem. #0 [ffffc90009b47b40] __schedule+700 at ffffffff818667cc #1 [ffffc90009b47bd8] schedule+54 at ffffffff81866de6 #2 [ffffc90009b47bf0] rwsem_down_write_failed+536 at ffffffff8186aa28 #3 [ffffc90009b47c88] call_rwsem_down_write_failed+23 at ffffffff8185a1b7 #4 [ffffc90009b47cd0] down_write+45 at ffffffff81869c9d #5 [ffffc90009b47ce8] ocfs2_dio_end_io_write+180 at ffffffffc05d5444 [ocfs2] #6 [ffffc90009b47dd8] ocfs2_dio_end_io+85 at ffffffffc05d5a85 [ocfs2] #7 [ffffc90009b47e18] dio_complete+140 at ffffffff812c873c #8 [ffffc90009b47e50] dio_aio_complete_work+25 at ffffffff812c89f9 #9 [ffffc90009b47e60] process_one_work+361 at ffffffff810b1889 #10 [ffffc90009b47ea8] worker_thread+77 at ffffffff810b233d #11 [ffffc90009b47f08] kthread+261 at ffffffff810b7fd5 #12 [ffffc90009b47f50] ret_from_fork+62 at ffffffff81a0035e Thus above forms ABBA deadlock. The same deadlock was mentioned in upstream commit 28f5a8a7c033cbf3e32277f4cc9c6afd74f05300. well, it seems that that commit just removed cluster lock (the victim of above dead lock) from the ABBA deadlock party. End-user visible effects: Process hang in truncate -> ocfs2_setattr path and other processes hang at ocfs2_dio_end_io_write path. This is to fix the deadlock itself. It removes inode_lock() call from dio completion path to remove the deadlock and add ip_alloc_sem lock in setattr path to synchronize the inode modifications. [wen.gang.wang@oracle.com: remove the "had_alloc_lock" as suggested] Link: https://lkml.kernel.org/r/20210402171344.1605-1-wen.gang.wang@oracle.com Link: https://lkml.kernel.org/r/20210331203654.3911-1-wen.gang.wang@oracle.com Signed-off-by: Wengang Wang Reviewed-by: Joseph Qi Cc: Mark Fasheh Cc: Joel Becker Cc: Junxiao Bi Cc: Changwei Ge Cc: Gang He Cc: Jun Piao Cc: Signed-off-by: Andrew Morton --- fs/ocfs2/aops.c | 11 +---------- fs/ocfs2/file.c | 8 ++++++-- 2 files changed, 7 insertions(+), 12 deletions(-) --- a/fs/ocfs2/aops.c~ocfs2-fix-deadlock-between-setattr-and-dio_end_io_write +++ a/fs/ocfs2/aops.c @@ -2295,7 +2295,7 @@ static int ocfs2_dio_end_io_write(struct struct ocfs2_alloc_context *meta_ac = NULL; handle_t *handle = NULL; loff_t end = offset + bytes; - int ret = 0, credits = 0, locked = 0; + int ret = 0, credits = 0; ocfs2_init_dealloc_ctxt(&dealloc); @@ -2306,13 +2306,6 @@ static int ocfs2_dio_end_io_write(struct !dwc->dw_orphaned) goto out; - /* ocfs2_file_write_iter will get i_mutex, so we need not lock if we - * are in that context. */ - if (dwc->dw_writer_pid != task_pid_nr(current)) { - inode_lock(inode); - locked = 1; - } - ret = ocfs2_inode_lock(inode, &di_bh, 1); if (ret < 0) { mlog_errno(ret); @@ -2393,8 +2386,6 @@ out: if (meta_ac) ocfs2_free_alloc_context(meta_ac); ocfs2_run_deallocs(osb, &dealloc); - if (locked) - inode_unlock(inode); ocfs2_dio_free_write_ctx(inode, dwc); return ret; --- a/fs/ocfs2/file.c~ocfs2-fix-deadlock-between-setattr-and-dio_end_io_write +++ a/fs/ocfs2/file.c @@ -1245,22 +1245,24 @@ int ocfs2_setattr(struct user_namespace goto bail_unlock; } } + down_write(&OCFS2_I(inode)->ip_alloc_sem); handle = ocfs2_start_trans(osb, OCFS2_INODE_UPDATE_CREDITS + 2 * ocfs2_quota_trans_credits(sb)); if (IS_ERR(handle)) { status = PTR_ERR(handle); mlog_errno(status); - goto bail_unlock; + goto bail_unlock_alloc; } status = __dquot_transfer(inode, transfer_to); if (status < 0) goto bail_commit; } else { + down_write(&OCFS2_I(inode)->ip_alloc_sem); handle = ocfs2_start_trans(osb, OCFS2_INODE_UPDATE_CREDITS); if (IS_ERR(handle)) { status = PTR_ERR(handle); mlog_errno(status); - goto bail_unlock; + goto bail_unlock_alloc; } } @@ -1273,6 +1275,8 @@ int ocfs2_setattr(struct user_namespace bail_commit: ocfs2_commit_trans(osb, handle); +bail_unlock_alloc: + up_write(&OCFS2_I(inode)->ip_alloc_sem); bail_unlock: if (status && inode_locked) { ocfs2_inode_unlock_tracker(inode, 1, &oh, had_lock);