From patchwork Wed Dec 16 16:49:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 11978015 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D15EC0018C for ; Wed, 16 Dec 2020 16:50:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3CCF6233F9 for ; Wed, 16 Dec 2020 16:50:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726839AbgLPQuU (ORCPT ); Wed, 16 Dec 2020 11:50:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33794 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726786AbgLPQuU (ORCPT ); Wed, 16 Dec 2020 11:50:20 -0500 Received: from mail-qt1-x82e.google.com (mail-qt1-x82e.google.com [IPv6:2607:f8b0:4864:20::82e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A9F26C0617A6 for ; Wed, 16 Dec 2020 08:49:39 -0800 (PST) Received: by mail-qt1-x82e.google.com with SMTP id y15so17666587qtv.5 for ; Wed, 16 Dec 2020 08:49:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=hSgyk6WFKYPfFgTIZUe26+LsjxefCswJce9lzbwUkiA=; b=FCkaJs5pazo9YE2TevtrlyDEhs5UCSuHdQTMMSCjP7/T3oZ03P6hNG4lPTmHT3QXWO eucoF9Fr/0Glp739UhwgTSleAhy0Mf6sKuEpQrLIwcYwvLfljbjDaXOh76oVFXr8bpTW V+t4DKa3Rwgh5tKqLhqrkH3I6IIkZ6n22wAVJppzpBytv+mQqXnDsJvLAuaGcdB1sARj g8LPOaWf7+eU7huVUNnvor2OSd0YOtwO1gL/gAZOOUC4hEsfJXqLPHTFPyUt6pXRzEv1 ltwjbnMwBnIgD1q2BruW0jOTwgJCb0I1w3nvZagIQPaJoqzb3TnMH4p6oUjGywevIgQj UZJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=hSgyk6WFKYPfFgTIZUe26+LsjxefCswJce9lzbwUkiA=; b=o2tEP1RGn7HQXadFSayo/Nrleu/VBQZliCJLLi8G2DIPmjcdByAbGSMipKMM0MO5Jm yuN9wyAK8S2zKdQepDwEWFEJpo7j9aJJ68EoQEHGdVyqC21UPPZlRU+36lJZaxvvLOOL 4TfNUhtB3f9zjmoYIEC8znj9N0g8k4FlSFckjinrOHDhhBkvDyKorGddSVc1gl8DuuJe NOc5qGUKuXqVX9ISmllYi/KPzQ85f+4JOwsPjdJtwvEPSxLEoIKAADOWmud7+7dkYhmF rlTQfXT/9R1fpNHtJDyduI0dvk3pxOb0R0YCrL6LCSiFMuvRSBhE8kczgMRGQpMSqrzu FyGg== X-Gm-Message-State: AOAM532QTFq9Tf18vHp88nFne+V9RhifxmUOaI0Z8MzIH/nj9R8aqg+M tKjdGxxMQp3nyfLYL0PZmsmti4PvL2Vc8rrf X-Google-Smtp-Source: ABdhPJwAwzQrzrOQaQXKCfltAP4zH2UfukJTnnJjumEKvSQ2z5L2h9K4X1ZA1XYqT92lDXjnasDJRw== X-Received: by 2002:ac8:44d8:: with SMTP id b24mr42523179qto.339.1608137378606; Wed, 16 Dec 2020 08:49:38 -0800 (PST) Received: from localhost (cpe-174-109-172-136.nc.res.rr.com. [174.109.172.136]) by smtp.gmail.com with ESMTPSA id b12sm1272029qtj.12.2020.12.16.08.49.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Dec 2020 08:49:38 -0800 (PST) From: Josef Bacik To: linux-btrfs@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v3 1/6] btrfs: do not block on deleted bgs mutex in the cleaner Date: Wed, 16 Dec 2020 11:49:29 -0500 Message-Id: <97ae17159b0c41e04630bf5449df91d0a1993f2e.1608137316.git.josef@toxicpanda.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org While running some stress tests I started getting hung task messages. This is because the delete unused bg's code has to take the delete_unused_bgs_mutex to do it's work, which is taken by balance to make sure we don't delete block groups while we're balancing. The problem is a balance can take a while, and so we were getting hung task warnings. We don't need to block and run these things, and the cleaner is needed to do other work, so trylock on this mutex and just bail if we can't acquire it right away. Signed-off-by: Josef Bacik --- fs/btrfs/block-group.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/fs/btrfs/block-group.c b/fs/btrfs/block-group.c index 52f2198d44c9..f61e275bd792 100644 --- a/fs/btrfs/block-group.c +++ b/fs/btrfs/block-group.c @@ -1254,6 +1254,13 @@ void btrfs_delete_unused_bgs(struct btrfs_fs_info *fs_info) if (!test_bit(BTRFS_FS_OPEN, &fs_info->flags)) return; + /* + * Long running balances can keep us blocked here for eternity, so + * simply skip deletion if we're unable to get the mutex. + */ + if (!mutex_trylock(&fs_info->delete_unused_bgs_mutex)) + return; + spin_lock(&fs_info->unused_bgs_lock); while (!list_empty(&fs_info->unused_bgs)) { int trimming; @@ -1273,8 +1280,6 @@ void btrfs_delete_unused_bgs(struct btrfs_fs_info *fs_info) btrfs_discard_cancel_work(&fs_info->discard_ctl, block_group); - mutex_lock(&fs_info->delete_unused_bgs_mutex); - /* Don't want to race with allocators so take the groups_sem */ down_write(&space_info->groups_sem); @@ -1420,11 +1425,11 @@ void btrfs_delete_unused_bgs(struct btrfs_fs_info *fs_info) end_trans: btrfs_end_transaction(trans); next: - mutex_unlock(&fs_info->delete_unused_bgs_mutex); btrfs_put_block_group(block_group); spin_lock(&fs_info->unused_bgs_lock); } spin_unlock(&fs_info->unused_bgs_lock); + mutex_unlock(&fs_info->delete_unused_bgs_mutex); return; flip_async: From patchwork Wed Dec 16 16:49:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 11978017 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8AFF6C2BBCD for ; Wed, 16 Dec 2020 16:50:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5969C233F8 for ; Wed, 16 Dec 2020 16:50:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726846AbgLPQuW (ORCPT ); Wed, 16 Dec 2020 11:50:22 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33802 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726786AbgLPQuW (ORCPT ); Wed, 16 Dec 2020 11:50:22 -0500 Received: from mail-qk1-x729.google.com (mail-qk1-x729.google.com [IPv6:2607:f8b0:4864:20::729]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5F5C6C0617A7 for ; Wed, 16 Dec 2020 08:49:41 -0800 (PST) Received: by mail-qk1-x729.google.com with SMTP id 186so23146738qkj.3 for ; Wed, 16 Dec 2020 08:49:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9CeSOnTCiDSiiYTmZsB59g/wxyjTSvdjd2ghnfRV0Lo=; b=vTwJdAL/gSVT897eY8kuccI4Vlw9YfRTe5JPnll1PVjR7/U8yvVQnDdx+2gPEbmnKH dPB9FD8UbTn0ctjYVdnR14zR7SPZeSjPYWqPrANxROerpPtY74htaPDzgAW42kllrhjS o0IpjW7/qdYsXiydZsbbNa0uaXmqt+LZnyCFtS+MibZzrOgkEqKPiF0ETqe805YbEWe6 QQY5ZEM8uUP6QEmGGxOVy+llKUo1Xn5nvZwzCsC2jvHYn7TJvJpx/9IeuQykfm0oFE96 hjtDK6IbkWbr/SswWKZALyGViX+7+NpUrrwNLU8uVJA+ooGGMvvPvaQmRksjxTvgmRLr sabA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9CeSOnTCiDSiiYTmZsB59g/wxyjTSvdjd2ghnfRV0Lo=; b=qroM7yzcwIaou4m9V5Jh6kRtGjJpN0Z4q2FQkvR7NIfsy4pE4IhFB4vKxMXoLoCdyJ ogWfgc6sMOxPc4gCjLeTxHuaeV9Hd6q8TgHV5l0gYCHiUL7unx5I927OjyjT5/dN0cP4 JhNWK6bpi7Y/XJBJVDuiLx6EpbBskvBlkW5YZYmI3x+UsKHxtbg4gPLP212oiVg9faIw W6kRvdsh5TSlhCSP0rwaZyRL8dUr8wTihZKgqCLjZRca9GhYqgfQarPXg76v+1Tu21FU fwMupEDuKXDSfzsOkQW7ZS8mcgF1iSGrE9i7KSPIcy1DBNDdremMDdPS8j10hdpcPO4r iphQ== X-Gm-Message-State: AOAM532n7C0no2ZX0V34aQlLxyAvzI6dH7lAZlbFjb5MJSuGB+EMg+W7 O/4QR4EImf4HYQOSeC8SsROaLfFQYgyvDzmv X-Google-Smtp-Source: ABdhPJzK46mG2BSHhTWnu+fd5n6CcRlZDW9nge+oJcbEF0ymg0uGs0lZ2crQSyhrSCU+gJYdyh53TA== X-Received: by 2002:a37:6c1:: with SMTP id 184mr42617755qkg.381.1608137380271; Wed, 16 Dec 2020 08:49:40 -0800 (PST) Received: from localhost (cpe-174-109-172-136.nc.res.rr.com. [174.109.172.136]) by smtp.gmail.com with ESMTPSA id h26sm1270164qtq.18.2020.12.16.08.49.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Dec 2020 08:49:39 -0800 (PST) From: Josef Bacik To: linux-btrfs@vger.kernel.org, kernel-team@fb.com Cc: Nikolay Borisov Subject: [PATCH v3 2/6] btrfs: only let one thread pre-flush delayed refs in commit Date: Wed, 16 Dec 2020 11:49:30 -0500 Message-Id: <22da6e5f334b5ec148147c8382033e849cf0d7e9.1608137316.git.josef@toxicpanda.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org I've been running a stress test that runs 20 workers in their own subvolume, which are running an fsstress instance with 4 threads per worker, which is 80 total fsstress threads. In addition to this I'm running balance in the background as well as creating and deleting snapshots. This test takes around 12 hours to run normally, going slower and slower as the test goes on. The reason for this is because fsstress is running fsync sometimes, and because we're messing with block groups we often fall through to btrfs_commit_transaction, so will often have 20-30 threads all calling btrfs_commit_transaction at the same time. These all get stuck contending on the extent tree while they try to run delayed refs during the initial part of the commit. This is suboptimal, really because the extent tree is a single point of failure we only want one thread acting on that tree at once to reduce lock contention. Fix this by making the flushing mechanism a bit operation, to make it easy to use test_and_set_bit() in order to make sure only one task does this initial flush. Once we're into the transaction commit we only have one thread doing delayed ref running, it's just this initial pre-flush that is problematic. With this patch my stress test takes around 90 minutes to run, instead of 12 hours. Reviewed-by: Nikolay Borisov Signed-off-by: Josef Bacik --- fs/btrfs/delayed-ref.h | 12 ++++++------ fs/btrfs/transaction.c | 32 ++++++++++++++++---------------- 2 files changed, 22 insertions(+), 22 deletions(-) diff --git a/fs/btrfs/delayed-ref.h b/fs/btrfs/delayed-ref.h index 1c977e6d45dc..6e414785b56f 100644 --- a/fs/btrfs/delayed-ref.h +++ b/fs/btrfs/delayed-ref.h @@ -135,6 +135,11 @@ struct btrfs_delayed_data_ref { u64 offset; }; +enum btrfs_delayed_ref_flags { + /* Used to indicate that we are flushing delayed refs for the commit. */ + BTRFS_DELAYED_REFS_FLUSHING, +}; + struct btrfs_delayed_ref_root { /* head ref rbtree */ struct rb_root_cached href_root; @@ -158,12 +163,7 @@ struct btrfs_delayed_ref_root { u64 pending_csums; - /* - * set when the tree is flushing before a transaction commit, - * used by the throttling code to decide if new updates need - * to be run right away - */ - int flushing; + unsigned long flags; u64 run_delayed_start; diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c index f51f9e39bcee..9c39b5c3f0fc 100644 --- a/fs/btrfs/transaction.c +++ b/fs/btrfs/transaction.c @@ -911,7 +911,8 @@ bool btrfs_should_end_transaction(struct btrfs_trans_handle *trans) smp_mb(); if (cur_trans->state >= TRANS_STATE_COMMIT_START || - cur_trans->delayed_refs.flushing) + test_bit(BTRFS_DELAYED_REFS_FLUSHING, + &cur_trans->delayed_refs.flags)) return true; return should_end_transaction(trans); @@ -2043,23 +2044,22 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans) btrfs_trans_release_metadata(trans); trans->block_rsv = NULL; - /* make a pass through all the delayed refs we have so far - * any runnings procs may add more while we are here - */ - ret = btrfs_run_delayed_refs(trans, 0); - if (ret) { - btrfs_end_transaction(trans); - return ret; - } - - cur_trans = trans->transaction; - /* - * set the flushing flag so procs in this transaction have to - * start sending their work down. + * We only want one transaction commit doing the flushing so we do not + * waste a bunch of time on lock contention on the extent root node. */ - cur_trans->delayed_refs.flushing = 1; - smp_wmb(); + if (!test_and_set_bit(BTRFS_DELAYED_REFS_FLUSHING, + &cur_trans->delayed_refs.flags)) { + /* + * make a pass through all the delayed refs we have so far + * any runnings procs may add more while we are here + */ + ret = btrfs_run_delayed_refs(trans, 0); + if (ret) { + btrfs_end_transaction(trans); + return ret; + } + } btrfs_create_pending_block_groups(trans); From patchwork Wed Dec 16 16:49:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 11978019 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A43A0C2BBD4 for ; Wed, 16 Dec 2020 16:50:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 72D9D233FA for ; Wed, 16 Dec 2020 16:50:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726851AbgLPQuZ (ORCPT ); Wed, 16 Dec 2020 11:50:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33810 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726786AbgLPQuZ (ORCPT ); Wed, 16 Dec 2020 11:50:25 -0500 Received: from mail-qk1-x72f.google.com (mail-qk1-x72f.google.com [IPv6:2607:f8b0:4864:20::72f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AB08DC0617B0 for ; Wed, 16 Dec 2020 08:49:43 -0800 (PST) Received: by mail-qk1-x72f.google.com with SMTP id d14so22473607qkc.13 for ; Wed, 16 Dec 2020 08:49:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=eIfUdd9SMDMV7st/xib0XwZ+96qQemHgJNdz3/SGt4o=; b=mEdvCku8cUTU4ytkdx5/xTyVHsgf5crSy8V0g14UjjJjpbAZLOZ25NrH8h8gMmJ1Wh hsf45dDXjPAKrQVjyVOGVmzQ5aRBMmFlCkYg0hLF3ufBf3xDW117H4Eg0GHYwMx51SpU NjAghfY27TfB2Mqp/eH/xds0rv3cuLjuO1WhNUmnay/+t1d0DRzQ9yqE4z3L+oEo0ERI dsrkcIUihAjO3GJ4sdDvA949J2mCIv6LDZ8+6q/GUsTbxLvRH/8TiVE+A2RezFUL8CCp jaRYp5fvBaNwCMDuO53chpF8+gA78ZkZWOpmdyNFyULifSCFR9Ax8g589p0AL+TcOGs7 NP/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=eIfUdd9SMDMV7st/xib0XwZ+96qQemHgJNdz3/SGt4o=; b=TftXxB0AMARqxIyXgczyG/G44Gyg+1Ny24eY1qiirSd2EWP6opm9MAAytqrfMJXn+b ZnrZ5l4kGy41xMMIZIMaX9Up4zpUQj+OqUR+e3cxA33hFkHee+Rrz9oveH/MvdDN+K3v jzh89083GdUG5Xvm2xCuwg7MluhJusOMUIwPDiib/26LAjDLj3dvlhzyR8Zw05l3p1aO Dy4DFsRHRx86zWtx++Y45i2+vmeuStnNQMAT7/hM59pyhiDxl42oyTeL/oGQv3+miGuZ 9mC7oETkxr7iq1tAnh1UiOI6qzGraRip0SRQs/nvIOOcLc68WIDJzmH2Z4jjeCTC5rQR fGxw== X-Gm-Message-State: AOAM5318bfIfSTqUm+viiIhv0uoS7Mo1CNBzy126p6DPgb0Ii/u6TvCn RCAa7yHjIyOHp8MiMLHq0+ZdOxu+jSXiVzaA X-Google-Smtp-Source: ABdhPJx27wTtrnuCqfqbrYn0atKoooWzbQFvNOVe+TVzw0T9ITw6nyxBKHyZgu7DtUZSHRKMkiSGEg== X-Received: by 2002:a05:620a:126a:: with SMTP id b10mr3941842qkl.354.1608137381908; Wed, 16 Dec 2020 08:49:41 -0800 (PST) Received: from localhost (cpe-174-109-172-136.nc.res.rr.com. [174.109.172.136]) by smtp.gmail.com with ESMTPSA id 190sm1559030qkf.61.2020.12.16.08.49.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Dec 2020 08:49:41 -0800 (PST) From: Josef Bacik To: linux-btrfs@vger.kernel.org, kernel-team@fb.com Cc: Nikolay Borisov Subject: [PATCH v3 3/6] btrfs: delayed refs pre-flushing should only run the heads we have Date: Wed, 16 Dec 2020 11:49:31 -0500 Message-Id: X-Mailer: git-send-email 2.26.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org Previously our delayed ref running used the total number of items as the items to run. However we changed that to number of heads to run with the delayed_refs_rsv, as generally we want to run all of the operations for one bytenr. But with btrfs_run_delayed_refs(trans, 0) we set our count to 2x the number of items that we have. This is generally fine, but if we have some operation generation loads of delayed refs while we're doing this pre-flushing in the transaction commit, we'll just spin forever doing delayed refs. Fix this to simply pick the number of delayed refs we currently have, that way we do not end up doing a lot of extra work that's being generated in other threads. Signed-off-by: Josef Bacik Reviewed-by: Nikolay Borisov --- fs/btrfs/extent-tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index d79b8369e6aa..b6d774803a2c 100644 --- a/fs/btrfs/extent-tree.c +++ b/fs/btrfs/extent-tree.c @@ -2160,7 +2160,7 @@ int btrfs_run_delayed_refs(struct btrfs_trans_handle *trans, delayed_refs = &trans->transaction->delayed_refs; if (count == 0) - count = atomic_read(&delayed_refs->num_entries) * 2; + count = delayed_refs->num_heads_ready; again: #ifdef SCRAMBLE_DELAYED_REFS From patchwork Wed Dec 16 16:49:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 11978021 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07511C0018C for ; Wed, 16 Dec 2020 16:50:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BC7DA233F8 for ; Wed, 16 Dec 2020 16:50:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726854AbgLPQui (ORCPT ); Wed, 16 Dec 2020 11:50:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33846 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726786AbgLPQui (ORCPT ); Wed, 16 Dec 2020 11:50:38 -0500 Received: from mail-qk1-x730.google.com (mail-qk1-x730.google.com [IPv6:2607:f8b0:4864:20::730]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5C26CC06138C for ; Wed, 16 Dec 2020 08:49:46 -0800 (PST) Received: by mail-qk1-x730.google.com with SMTP id p14so14027501qke.6 for ; Wed, 16 Dec 2020 08:49:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=dVXO8pW/I15AKnv0A8Q8X6v1okWxT+hR89oXARRUey8=; b=e3aFwPO+g2BFM03R2174BIVWvLzaB88bIwZHQ6AQ4JspXMUNjBVN96Tx7aZShCx/f7 e+qLaH2v1WsrHFXOJb+Za/dy3DVB4TzUSb5L6hySVwJfKmRanwUoqUsriWY7PCsr8CUO +THtRqTEsrahKv488ty6zL6fDc2Pc5DvOL+XhJo8sZ+jGAE5SQ/M67zRfpMgl2N4f+2k xHpgFQ5PRe6SE3wNToWTYxy5NhvrF7CJPRYE5CuRb7pzhFlstPZ39LlOMCjCG6Grj56H m1UcTcM9zI5uoTrhCtyq9RXIph+H7xwp0KPzloLciLBLb1alkNF+yIctiiTWK6HwjIFf kFBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=dVXO8pW/I15AKnv0A8Q8X6v1okWxT+hR89oXARRUey8=; b=T2YKWC7dwFijPlV1QbM1EXnqgPgcHrFDWaxc6VDSwpLuS3v8rqwSTsrQu6ihFk2kLP CZ47eFt6ZM2DEQ7WtGeUpMjLG4KQeDK7nQ3I2EuLSkcTJqbwliNrkvfS+CFnsx/OQR8U 4mnt9Xuocal+gco5IMRsA3XrGay4rtlTuSWOiIFUAAqk9MatRfryFnHSJKQcQ9Gwdtm8 xe6fU0E2vRLHe3fjL20HbGQbxSq34fT+xNuVHg3WwZqjE1y309xYQPSPenD88fwIyLVv uRQfIC2Pxjen5VGLXKXaNPWiHL03R3jPNAY73zeqob8s0R1o8KpyyP+bgAC/7gwXLBmq smZw== X-Gm-Message-State: AOAM53171sZ1LJ0tQ4vx1BPUy0wj+aOT3bhUutEbKS+amH1EMM4CwTWC UgRFsrgHAFsZG5LQGPlQmIQlqsAUoiNuKpWB X-Google-Smtp-Source: ABdhPJxxmm3X+ueVHykRirqK28g85jo4q4PC6iaf98JokzYaDRrk8FtqEoNYpLEW+FEOsEX+wskveA== X-Received: by 2002:ae9:ddc3:: with SMTP id r186mr45215354qkf.452.1608137384473; Wed, 16 Dec 2020 08:49:44 -0800 (PST) Received: from localhost (cpe-174-109-172-136.nc.res.rr.com. [174.109.172.136]) by smtp.gmail.com with ESMTPSA id v5sm1424130qkv.64.2020.12.16.08.49.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Dec 2020 08:49:43 -0800 (PST) From: Josef Bacik To: linux-btrfs@vger.kernel.org, kernel-team@fb.com Cc: Nikolay Borisov Subject: [PATCH v3 4/6] btrfs: only run delayed refs once before committing Date: Wed, 16 Dec 2020 11:49:32 -0500 Message-Id: X-Mailer: git-send-email 2.26.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org We try to pre-flush the delayed refs when committing, because we want to do as little work as possible in the critical section of the transaction commit. However doing this twice can lead to very long transaction commit delays as other threads are allowed to continue to generate more delayed refs, which potentially delays the commit by multiple minutes in very extreme cases. So simply stick to one pre-flush, and then continue the rest of the transaction commit. Signed-off-by: Josef Bacik Reviewed-by: Nikolay Borisov --- fs/btrfs/transaction.c | 6 ------ 1 file changed, 6 deletions(-) diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c index 9c39b5c3f0fc..349e42300d2d 100644 --- a/fs/btrfs/transaction.c +++ b/fs/btrfs/transaction.c @@ -2063,12 +2063,6 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans) btrfs_create_pending_block_groups(trans); - ret = btrfs_run_delayed_refs(trans, 0); - if (ret) { - btrfs_end_transaction(trans); - return ret; - } - if (!test_bit(BTRFS_TRANS_DIRTY_BG_RUN, &cur_trans->flags)) { int run_it = 0; From patchwork Wed Dec 16 16:49:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 11978023 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6FDB5C2BBCA for ; Wed, 16 Dec 2020 16:50:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2DE1A233F8 for ; Wed, 16 Dec 2020 16:50:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726855AbgLPQui (ORCPT ); Wed, 16 Dec 2020 11:50:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33848 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726852AbgLPQui (ORCPT ); Wed, 16 Dec 2020 11:50:38 -0500 Received: from mail-qk1-x72a.google.com (mail-qk1-x72a.google.com [IPv6:2607:f8b0:4864:20::72a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B7EFEC061282 for ; Wed, 16 Dec 2020 08:49:47 -0800 (PST) Received: by mail-qk1-x72a.google.com with SMTP id d14so22473954qkc.13 for ; Wed, 16 Dec 2020 08:49:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=fwEdq8DUXBp6WdRinlQz1Q/HCzm/olXUBoqqxfXlrZg=; b=zLkqhE//XR5j1oGQwrMx1V5IbsMYyuM9/QqCtbHhNJnVEDiOsa6qy8b2PbPNLPXvLP OfGYhOCjMzb/cnlyrhTG9TwizDU+lmFjKRKy2PxlXKc8YpPacXDFg7rjkfquZUBbKSQr UC7pVjfdGns5cQrHNX99P9v82NSvzrBVitX6lps7a1DWSbXWxcRaBacvogTP2IthVHZ7 T/FGGcjQOWGzRz7aXJdnZy2AaUpr96ErtZut3m5PNQxZBrddwYN6bsEHomgcOAJdQB4V 4PbIaBZJu3tx8UXGax95p/+mXm04jXBxNOwfe2e0Hu3WNVBJExUg5QLGNarOLR0v+NLj 89Tw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fwEdq8DUXBp6WdRinlQz1Q/HCzm/olXUBoqqxfXlrZg=; b=qM7yOzm2aEF2o+pIs35EHFVIzrBmKFZ+QOjSTyMQJMnGEPJHvQPajwnEUBFYDbyRfz PJ6px5JbSK1krQFqmx5eC7UC3GlpvlMmm1DdIw1mUBP1LtvrAFrChZTega/rc6s8gTTn YZXur5Pu2umYYPlfvTIsMKS5NPXI4ToZ2K5zF0GPXY0lMf9HfuOTsEw7vIdXiNGIxRw8 dSSJ+tH14I/r0PltdoLVRNZ8t4rIqFpmI1Dw4d2r8qbWyYmMFscypmRe72EHTjV6wsU1 h/2XyupHDYROrhRiwSdWnvC3mIRcjI5udZxA1ySftND08pECvBpKWnPT0YKty4O+JWvB ty1w== X-Gm-Message-State: AOAM533KE3Xq4H1Q3iyggo4cSSrKn80NgCmkQIuaOw3/iT3LuSanncs6 FCX3BNkg8DAFiSNgHd56yK5bDwBQ//hF0hbm X-Google-Smtp-Source: ABdhPJyNx17XEljBqU1K3t9hW2AktmU+7tababkraG/uVt5C6BxFxaZdDSyCJNuFvP5+FYXRM+3XtQ== X-Received: by 2002:a37:8c42:: with SMTP id o63mr44307368qkd.12.1608137386661; Wed, 16 Dec 2020 08:49:46 -0800 (PST) Received: from localhost (cpe-174-109-172-136.nc.res.rr.com. [174.109.172.136]) by smtp.gmail.com with ESMTPSA id f134sm1457272qke.23.2020.12.16.08.49.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Dec 2020 08:49:46 -0800 (PST) From: Josef Bacik To: linux-btrfs@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v3 5/6] btrfs: run delayed refs less often in commit_cowonly_roots Date: Wed, 16 Dec 2020 11:49:33 -0500 Message-Id: <6130299d16abcd5d0e0aeb51bad2ff32e41deb25.1608137316.git.josef@toxicpanda.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org We love running delayed refs in commit_cowonly_roots, but it is a bit excessive. I was seeing cases of running 3 or 4 refs a few times in a row during this time. Instead simply update all of the roots first, then run delayed refs, then handle the empty block groups case, and then if we have any more dirty roots do the whole thing again. This allows us to be much more efficient with our delayed ref running, as we can batch a few more operations at once. Signed-off-by: Josef Bacik --- fs/btrfs/transaction.c | 12 ------------ 1 file changed, 12 deletions(-) diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c index 349e42300d2d..51c5c2f6e064 100644 --- a/fs/btrfs/transaction.c +++ b/fs/btrfs/transaction.c @@ -1687,12 +1687,6 @@ static noinline int create_pending_snapshot(struct btrfs_trans_handle *trans, goto fail; } - ret = btrfs_run_delayed_refs(trans, (unsigned long)-1); - if (ret) { - btrfs_abort_transaction(trans, ret); - goto fail; - } - /* * Do special qgroup accounting for snapshot, as we do some qgroup * snapshot hack to do fast snapshot. @@ -1740,12 +1734,6 @@ static noinline int create_pending_snapshot(struct btrfs_trans_handle *trans, } } - ret = btrfs_run_delayed_refs(trans, (unsigned long)-1); - if (ret) { - btrfs_abort_transaction(trans, ret); - goto fail; - } - fail: pending->error = ret; dir_item_existed: From patchwork Wed Dec 16 16:49:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 11978025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6EA83C4361B for ; Wed, 16 Dec 2020 16:50:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 402EA233F8 for ; Wed, 16 Dec 2020 16:50:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726852AbgLPQu6 (ORCPT ); Wed, 16 Dec 2020 11:50:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33904 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726830AbgLPQu6 (ORCPT ); Wed, 16 Dec 2020 11:50:58 -0500 Received: from mail-qk1-x72c.google.com (mail-qk1-x72c.google.com [IPv6:2607:f8b0:4864:20::72c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4460FC061285 for ; Wed, 16 Dec 2020 08:49:50 -0800 (PST) Received: by mail-qk1-x72c.google.com with SMTP id b64so19137901qkc.12 for ; Wed, 16 Dec 2020 08:49:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=8gAbf/w+Rfm6OuF2tFIhcPW/oJc3NjJb7aciQ2l7xDI=; b=A6DApy+5kW4760NawwHX7kSvOHvIi8Irv7Td55dn0bYRRFSqZkQbSTgIO8dxYqI9rN yJJovmpPhSA/ooUJWM+8UIxPj3y5Ccvg2okUV/JBVOqyM9+x1mvg//8GYxDM15+TW4VW /EkklcuwaeYVW8TCNdUC8M0ABceJHfiS10IkDXcBbuYEMSARwoHlTdz+OGtenXlAzDdu stoA1jtTtMJrEV8PAASZ1+k38965HzE3kImdxn6lFR73DY0Ajfxu4rMEYeu/6o5y7jzG fiFsVRWUn9vKfYOZ0ZpQ4MIEUGPq55RsSiHVTxTikDkRkE7cAgaV4BOXVU9rQEhO5D9I JmGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8gAbf/w+Rfm6OuF2tFIhcPW/oJc3NjJb7aciQ2l7xDI=; b=jZGQYzEV6J06za/WikuhQwsaDfsP9sevoIgBnEVtwDLZvIRzUZEx31FdBRF4XGk5X9 yiOgihuEdVt6XJlQXL9i1niZohruNuHeHbcY2EwknttjrQoPZDinSiPk7x7EQ4m7FDbo UoOjAFOhpEkgEHafaU6RQhBkGwKkXVpLRGZ13OqnFEWtTCAj45zWKSI5gs6iCOJp9TnO jEgcdsF6tlIBzMK7bwPk7mrKE7I+efiTpE23DwiNamEPkxHgh/IZwFj/eWqn2OkmnYsM KGXgb9Z1qNJj5If91f92gEVWe8sV/Hut6NRdfjPEhefr3uAriu5m6nOTE91eYq8BzRUo aIww== X-Gm-Message-State: AOAM531dlrXFiKBpjHpEqU/r3zLDHHihufw3ZzqQb3/IFQMEY5CpJyJj b009hRG4Uau0fTEcwZenIWhWg2TxsY3QZZq/ X-Google-Smtp-Source: ABdhPJyFNE9PQCylPRrblRar8+XJZxE0Lytd8fyipVnlTtviIQwrwKLWxDkQ2UvbotXRz8qC/dV3fg== X-Received: by 2002:ae9:eb8b:: with SMTP id b133mr44772144qkg.399.1608137389170; Wed, 16 Dec 2020 08:49:49 -0800 (PST) Received: from localhost (cpe-174-109-172-136.nc.res.rr.com. [174.109.172.136]) by smtp.gmail.com with ESMTPSA id a10sm1448883qkk.52.2020.12.16.08.49.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Dec 2020 08:49:47 -0800 (PST) From: Josef Bacik To: linux-btrfs@vger.kernel.org, kernel-team@fb.com Subject: [PATCH v3 6/6] btrfs: stop running all delayed refs during snapshot Date: Wed, 16 Dec 2020 11:49:34 -0500 Message-Id: <3643159e2e5f04612b0a3197ea32bd8591243131.1608137316.git.josef@toxicpanda.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org This was added a very long time ago to work around issues with delayed refs and with qgroups. Both of these issues have since been properly fixed, so all this does is cause a lot of lock contention with anybody else who is running delayed refs. Signed-off-by: Josef Bacik --- fs/btrfs/transaction.c | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c index 51c5c2f6e064..70cbc15fc58a 100644 --- a/fs/btrfs/transaction.c +++ b/fs/btrfs/transaction.c @@ -1228,10 +1228,6 @@ static noinline int commit_cowonly_roots(struct btrfs_trans_handle *trans) btrfs_tree_unlock(eb); free_extent_buffer(eb); - if (ret) - return ret; - - ret = btrfs_run_delayed_refs(trans, (unsigned long)-1); if (ret) return ret; @@ -1249,10 +1245,6 @@ static noinline int commit_cowonly_roots(struct btrfs_trans_handle *trans) if (ret) return ret; - /* run_qgroups might have added some more refs */ - ret = btrfs_run_delayed_refs(trans, (unsigned long)-1); - if (ret) - return ret; again: while (!list_empty(&fs_info->dirty_cowonly_roots)) { struct btrfs_root *root; @@ -1267,15 +1259,24 @@ static noinline int commit_cowonly_roots(struct btrfs_trans_handle *trans) ret = update_cowonly_root(trans, root); if (ret) return ret; - ret = btrfs_run_delayed_refs(trans, (unsigned long)-1); - if (ret) - return ret; } + /* Now flush any delayed refs generated by updating all of the roots. */ + ret = btrfs_run_delayed_refs(trans, (unsigned long)-1); + if (ret) + return ret; + while (!list_empty(dirty_bgs) || !list_empty(io_bgs)) { ret = btrfs_write_dirty_block_groups(trans); if (ret) return ret; + + /* + * We're writing the dirty block groups, which could generate + * delayed refs, which could generate more dirty block groups, + * so we want to keep this flushing in this loop to make sure + * everything gets run. + */ ret = btrfs_run_delayed_refs(trans, (unsigned long)-1); if (ret) return ret;