From patchwork Mon Dec 3 15:20:29 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 10709797 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2354D17D5 for ; Mon, 3 Dec 2018 15:20:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 13A5E2B31A for ; Mon, 3 Dec 2018 15:20:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 103972B434; Mon, 3 Dec 2018 15:20:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 828322B31A for ; Mon, 3 Dec 2018 15:20:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726689AbeLCPUp (ORCPT ); Mon, 3 Dec 2018 10:20:45 -0500 Received: from mail-yb1-f195.google.com ([209.85.219.195]:33087 "EHLO mail-yb1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726549AbeLCPUp (ORCPT ); Mon, 3 Dec 2018 10:20:45 -0500 Received: by mail-yb1-f195.google.com with SMTP id f125so674840ybc.0 for ; Mon, 03 Dec 2018 07:20:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=pRCx2eP/l80aL+OLubW7i+4YyKvLhkbzZ3rfIZGfT4g=; b=EggwHEZUIePK54lc0IS05HffMMVGVyjTBzNhIhSjPAmLdnO/DpQJKFSS7FCxj6637X Z2WrDa1hK2PbyFV0cW+gF8zOOZ5SowVt+jil8XGiRMyVhgfmgOWQ5de0C4mkKqpii4gz v7liwSAEOKtiRu6osHoFs/DaozLEk7FY7ouCpAmvcxUbegOeP3kim2EcFyYXST1yJ41F d31uXhZwX0Y7BlvSzQ5jjDKjt2sgAimOdDIUtDyBIf2Kj7hrGRyO8U6D3cZZ3sSfIhnW s+mpGtSd01OCYYhFgQn+GWJ2r9J/b2l6S9tTECIRVWIWNTiwSMmSRNxHqehEBWs2qK2R YwqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=pRCx2eP/l80aL+OLubW7i+4YyKvLhkbzZ3rfIZGfT4g=; b=HmEoSw8Q6D7ZxcQYJnLB6Z1EmJJ8NAoYyPh0AU+4O8CkRjuuiINFVfjs5ruzwEa16m MLguX2POt6QajwZkqZZGWddcrR6itvqNl+a3yu91DHHu0EsiBUkdFiPOBkSRt+z3Bz9f pbJHMlt00cEFx/OaH/4F0hlJoM5DxYllmm4khWQcLCPOluVs9paoV/OWFx7Rn0Jpsbzj fKP0rxPjsgEqT3sz5b/eUpNrLgoBIYe3scF0g3ScfqxxPCdfbbK9DQY/hyOAFA0ppaaU d2cKMB55NA4wGrzfMJLc4IjlTB/juIq10zNhtt6V743m+4BVNMibKust0+RJUuK+O1mN OYiQ== X-Gm-Message-State: AA+aEWbOacZZy55tu1zF7YbUD21l8vcHZb9E/5aMWuYUH/NXc6bhEbyw oPq+61iP/0pnCleD0TZHEOBJtSMRyIc= X-Google-Smtp-Source: AFSGD/Urd+IaPybVu+11T53mgtCNXGC87CCm1Pkcuzr0PKiErLzsBkbxPrL3KBhg3O/xx2duO3yymA== X-Received: by 2002:a25:d050:: with SMTP id h77-v6mr14796785ybg.423.1543850443411; Mon, 03 Dec 2018 07:20:43 -0800 (PST) Received: from localhost ([107.15.81.208]) by smtp.gmail.com with ESMTPSA id l83sm4496666ywb.23.2018.12.03.07.20.41 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 03 Dec 2018 07:20:42 -0800 (PST) From: Josef Bacik To: linux-btrfs@vger.kernel.org, kernel-team@fb.com Cc: Josef Bacik Subject: [PATCH 01/10] btrfs: add btrfs_delete_ref_head helper Date: Mon, 3 Dec 2018 10:20:29 -0500 Message-Id: <20181203152038.21388-2-josef@toxicpanda.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20181203152038.21388-1-josef@toxicpanda.com> References: <20181203152038.21388-1-josef@toxicpanda.com> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Josef Bacik We do this dance in cleanup_ref_head and check_ref_cleanup, unify it into a helper and cleanup the calling functions. Signed-off-by: Josef Bacik Reviewed-by: Omar Sandoval Reviewed-by: Nikolay Borisov --- fs/btrfs/delayed-ref.c | 14 ++++++++++++++ fs/btrfs/delayed-ref.h | 3 ++- fs/btrfs/extent-tree.c | 22 +++------------------- 3 files changed, 19 insertions(+), 20 deletions(-) diff --git a/fs/btrfs/delayed-ref.c b/fs/btrfs/delayed-ref.c index 9301b3ad9217..b3e4c9fcb664 100644 --- a/fs/btrfs/delayed-ref.c +++ b/fs/btrfs/delayed-ref.c @@ -400,6 +400,20 @@ struct btrfs_delayed_ref_head *btrfs_select_ref_head( return head; } +void btrfs_delete_ref_head(struct btrfs_delayed_ref_root *delayed_refs, + struct btrfs_delayed_ref_head *head) +{ + lockdep_assert_held(&delayed_refs->lock); + lockdep_assert_held(&head->lock); + + rb_erase_cached(&head->href_node, &delayed_refs->href_root); + RB_CLEAR_NODE(&head->href_node); + atomic_dec(&delayed_refs->num_entries); + delayed_refs->num_heads--; + if (head->processing == 0) + delayed_refs->num_heads_ready--; +} + /* * Helper to insert the ref_node to the tail or merge with tail. * diff --git a/fs/btrfs/delayed-ref.h b/fs/btrfs/delayed-ref.h index 8e20c5cb5404..d2af974f68a1 100644 --- a/fs/btrfs/delayed-ref.h +++ b/fs/btrfs/delayed-ref.h @@ -261,7 +261,8 @@ static inline void btrfs_delayed_ref_unlock(struct btrfs_delayed_ref_head *head) { mutex_unlock(&head->mutex); } - +void btrfs_delete_ref_head(struct btrfs_delayed_ref_root *delayed_refs, + struct btrfs_delayed_ref_head *head); struct btrfs_delayed_ref_head *btrfs_select_ref_head( struct btrfs_delayed_ref_root *delayed_refs); diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index d242a1174e50..c36b3a42f2bb 100644 --- a/fs/btrfs/extent-tree.c +++ b/fs/btrfs/extent-tree.c @@ -2474,12 +2474,9 @@ static int cleanup_ref_head(struct btrfs_trans_handle *trans, spin_unlock(&delayed_refs->lock); return 1; } - delayed_refs->num_heads--; - rb_erase_cached(&head->href_node, &delayed_refs->href_root); - RB_CLEAR_NODE(&head->href_node); + btrfs_delete_ref_head(delayed_refs, head); spin_unlock(&head->lock); spin_unlock(&delayed_refs->lock); - atomic_dec(&delayed_refs->num_entries); trace_run_delayed_ref_head(fs_info, head, 0); @@ -6984,22 +6981,9 @@ static noinline int check_ref_cleanup(struct btrfs_trans_handle *trans, if (!mutex_trylock(&head->mutex)) goto out; - /* - * at this point we have a head with no other entries. Go - * ahead and process it. - */ - rb_erase_cached(&head->href_node, &delayed_refs->href_root); - RB_CLEAR_NODE(&head->href_node); - atomic_dec(&delayed_refs->num_entries); - - /* - * we don't take a ref on the node because we're removing it from the - * tree, so we just steal the ref the tree was holding. - */ - delayed_refs->num_heads--; - if (head->processing == 0) - delayed_refs->num_heads_ready--; + btrfs_delete_ref_head(delayed_refs, head); head->processing = 0; + spin_unlock(&head->lock); spin_unlock(&delayed_refs->lock); From patchwork Mon Dec 3 15:20:30 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 10709799 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4096814E2 for ; Mon, 3 Dec 2018 15:20:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 321552B24F for ; Mon, 3 Dec 2018 15:20:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 306AC2B42A; Mon, 3 Dec 2018 15:20:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A79952B25C for ; Mon, 3 Dec 2018 15:20:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726693AbeLCPUr (ORCPT ); Mon, 3 Dec 2018 10:20:47 -0500 Received: from mail-yw1-f66.google.com ([209.85.161.66]:36530 "EHLO mail-yw1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726549AbeLCPUq (ORCPT ); Mon, 3 Dec 2018 10:20:46 -0500 Received: by mail-yw1-f66.google.com with SMTP id y194so5524707ywg.3 for ; Mon, 03 Dec 2018 07:20:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=GhTGuLJR3TTmW9f1Z2iNY5zzX2M92cRLSuapIejz8FE=; b=b6UJ0E167ETa2nuCLPRFgHcMpopC1D9sg3QKAjFTrPF+vVlthTrv7aBX/mpWbag8sb FviFTCuMMMVcud7cG9s7H67J8Nb8cpmlGwQt5+WuGybe1RLD/iQZNHUQNyYbT54PXyCk bwykGz4BN2fxgxHU8gzz+wVn3uv++BwHapW7rKGOFjlF4kgNuUQkrjxWkpFkculDND9c gR025Fpt6itOTFfcIQQO+0K8cib5/J973rUYZMtDFf/3YvsTmLbdk5bNj5yIo+5AK4Ka Xp62NntngN2Agk3RHcGLkuy6ObaE58vP9PQ/Adg5+AYkwfrCF+VZwRi/hcOgDkE+VJEd ulNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=GhTGuLJR3TTmW9f1Z2iNY5zzX2M92cRLSuapIejz8FE=; b=G03ONGysW0n/KQ2ZiiId0CBTcLahzZh8nONgz6DEBNyYdcLRLmkCs5+nr1THMRNbU8 n1l5dt25xWYx/mdSFVFWSdeMUQfB3ibeDNhJHj1720mxcxjENNyNCRDI3u6OIuWAid0p cdDyi8YbabfAemTFYIWGXF67lJUJzjXVwZXauQXKXv0Pnq/r3l/WqFt42Y24t8FeOiNL sI/YgzyN11dB//mm2pWVl3xeLeeFu55zc0mdjyQySrcHFf9+mcTIzFv25iqxbce5QLuU QZueGR58JTAjsQg8unZSwss4ZQWgQGoFlC6Tm5ObUOwP7xjnI9QP1kh0tCPV+tEmu/4Y 6uUA== X-Gm-Message-State: AA+aEWYzC5gnTtQX0eWqup5nk0k6PgwBbKSUcsCvdFybWo+q5AepyWVL UgFIvwsuYj/emeYJZ1pvahWMonDfKn8= X-Google-Smtp-Source: AFSGD/Vj/9jgjg4H+IY4LVwERJip+HQESw5AiFq8m3vKSKVTGbieiJjUlzfd4wqpSyfji3ft2irY0w== X-Received: by 2002:a81:b64c:: with SMTP id h12mr16483692ywk.296.1543850444992; Mon, 03 Dec 2018 07:20:44 -0800 (PST) Received: from localhost ([107.15.81.208]) by smtp.gmail.com with ESMTPSA id m3sm4400917ywe.90.2018.12.03.07.20.44 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 03 Dec 2018 07:20:44 -0800 (PST) From: Josef Bacik To: linux-btrfs@vger.kernel.org, kernel-team@fb.com Cc: Josef Bacik Subject: [PATCH 02/10] btrfs: add cleanup_ref_head_accounting helper Date: Mon, 3 Dec 2018 10:20:30 -0500 Message-Id: <20181203152038.21388-3-josef@toxicpanda.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20181203152038.21388-1-josef@toxicpanda.com> References: <20181203152038.21388-1-josef@toxicpanda.com> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Josef Bacik We were missing some quota cleanups in check_ref_cleanup, so break the ref head accounting cleanup into a helper and call that from both check_ref_cleanup and cleanup_ref_head. This will hopefully ensure that we don't screw up accounting in the future for other things that we add. Reviewed-by: Omar Sandoval Reviewed-by: Liu Bo Signed-off-by: Josef Bacik Reviewed-by: Nikolay Borisov --- fs/btrfs/extent-tree.c | 67 +++++++++++++++++++++++++++++--------------------- 1 file changed, 39 insertions(+), 28 deletions(-) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index c36b3a42f2bb..e3ed3507018d 100644 --- a/fs/btrfs/extent-tree.c +++ b/fs/btrfs/extent-tree.c @@ -2443,6 +2443,41 @@ static int cleanup_extent_op(struct btrfs_trans_handle *trans, return ret ? ret : 1; } +static void cleanup_ref_head_accounting(struct btrfs_trans_handle *trans, + struct btrfs_delayed_ref_head *head) +{ + struct btrfs_fs_info *fs_info = trans->fs_info; + struct btrfs_delayed_ref_root *delayed_refs = + &trans->transaction->delayed_refs; + + if (head->total_ref_mod < 0) { + struct btrfs_space_info *space_info; + u64 flags; + + if (head->is_data) + flags = BTRFS_BLOCK_GROUP_DATA; + else if (head->is_system) + flags = BTRFS_BLOCK_GROUP_SYSTEM; + else + flags = BTRFS_BLOCK_GROUP_METADATA; + space_info = __find_space_info(fs_info, flags); + ASSERT(space_info); + percpu_counter_add_batch(&space_info->total_bytes_pinned, + -head->num_bytes, + BTRFS_TOTAL_BYTES_PINNED_BATCH); + + if (head->is_data) { + spin_lock(&delayed_refs->lock); + delayed_refs->pending_csums -= head->num_bytes; + spin_unlock(&delayed_refs->lock); + } + } + + /* Also free its reserved qgroup space */ + btrfs_qgroup_free_delayed_ref(fs_info, head->qgroup_ref_root, + head->qgroup_reserved); +} + static int cleanup_ref_head(struct btrfs_trans_handle *trans, struct btrfs_delayed_ref_head *head) { @@ -2478,31 +2513,6 @@ static int cleanup_ref_head(struct btrfs_trans_handle *trans, spin_unlock(&head->lock); spin_unlock(&delayed_refs->lock); - trace_run_delayed_ref_head(fs_info, head, 0); - - if (head->total_ref_mod < 0) { - struct btrfs_space_info *space_info; - u64 flags; - - if (head->is_data) - flags = BTRFS_BLOCK_GROUP_DATA; - else if (head->is_system) - flags = BTRFS_BLOCK_GROUP_SYSTEM; - else - flags = BTRFS_BLOCK_GROUP_METADATA; - space_info = __find_space_info(fs_info, flags); - ASSERT(space_info); - percpu_counter_add_batch(&space_info->total_bytes_pinned, - -head->num_bytes, - BTRFS_TOTAL_BYTES_PINNED_BATCH); - - if (head->is_data) { - spin_lock(&delayed_refs->lock); - delayed_refs->pending_csums -= head->num_bytes; - spin_unlock(&delayed_refs->lock); - } - } - if (head->must_insert_reserved) { btrfs_pin_extent(fs_info, head->bytenr, head->num_bytes, 1); @@ -2512,9 +2522,9 @@ static int cleanup_ref_head(struct btrfs_trans_handle *trans, } } - /* Also free its reserved qgroup space */ - btrfs_qgroup_free_delayed_ref(fs_info, head->qgroup_ref_root, - head->qgroup_reserved); + cleanup_ref_head_accounting(trans, head); + + trace_run_delayed_ref_head(fs_info, head, 0); btrfs_delayed_ref_unlock(head); btrfs_put_delayed_ref_head(head); return 0; @@ -6991,6 +7001,7 @@ static noinline int check_ref_cleanup(struct btrfs_trans_handle *trans, if (head->must_insert_reserved) ret = 1; + cleanup_ref_head_accounting(trans, head); mutex_unlock(&head->mutex); btrfs_put_delayed_ref_head(head); return ret; From patchwork Mon Dec 3 15:20:31 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 10709801 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9ABD117D5 for ; Mon, 3 Dec 2018 15:20:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 883A32B3C9 for ; Mon, 3 Dec 2018 15:20:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7C43B2B2AF; Mon, 3 Dec 2018 15:20:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0C9482B322 for ; Mon, 3 Dec 2018 15:20:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726697AbeLCPUt (ORCPT ); Mon, 3 Dec 2018 10:20:49 -0500 Received: from mail-yw1-f68.google.com ([209.85.161.68]:45272 "EHLO mail-yw1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726549AbeLCPUt (ORCPT ); Mon, 3 Dec 2018 10:20:49 -0500 Received: by mail-yw1-f68.google.com with SMTP id d190so5507157ywd.12 for ; Mon, 03 Dec 2018 07:20:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=bGA7flS0OJ7Wj4K/89zBSRUCCCpY4sVo2fIC68LsQr0=; b=QWKP7MoOy1DTuibmfMmvFNuWLKrDxfOPNphh3dK3VyGDFyUC2JZJLsHfLU+E7VGpT/ MPS2Sqpzn5f9YLfM2MNK8AwMUxNpgxJQgLcfP3BKz1sCMtw4/rrWFTIAJ94rjxmWPkiu 7Zk3R12WsH1Gp+Jibo9IxBhLuG1Cs1+O5oKWb4w/GvxdlwqFpZp5bZKUA8eNHfsylTy4 cuZas/9dNljeExDnGswcvjFqmQiGCNaBZL0SO/IlnbJ1gvNmM2YUsGeFk6bfq3/XU5kx cuR+aWpXavKbPYVwm79WE3MeyUM30/TXrvckKSlYwhd2A0OLZhtVvfBXzKy264dkP/es 9FGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=bGA7flS0OJ7Wj4K/89zBSRUCCCpY4sVo2fIC68LsQr0=; b=jt7R+E8mybSrXbHl3eaoXF7H+jtXe/FL9TusECIuiv7EPTaCbDiZqq3I55EznUoVzq 5XaJ4a/VUNmDe6YwAab51IbFLkrgTDnqn/i2BNMju59xuN9kIp8DepWaXLYjE68+bMVt CJROgDZZdpORZ5T6OM3nn9FE5+q2LQ0FEaK1fAFzEF5e9cq7natFQXHwGBjFj8E/qxRR Oz84z1oxxWvU6ghSCvCSJ+zO17M/4HQSAQhmqezdfUX6X0auEZYKNFPT4teBHOZbB4Bb N3OCkD0XhtZCGHROGKT6h+8DZIsPOsd2WlTEmeL2GYwjYl7zjK/ltc2T2RQEcVlJOF5M 6+Lw== X-Gm-Message-State: AA+aEWbnjDYEUMceghY3sAsfK8BWn/WXa3DK14wy0zlCiHcYJ4lkPuth 05htCP5GbPvzgBin6sJM6uoY7+kbk2o= X-Google-Smtp-Source: AFSGD/WBb099OZQAPI0SNhTxrH/hjVO6jQp3uSPwWwLFNVzleQGpRrJsxBUS60qgt1VaZkaE/HZ06g== X-Received: by 2002:a81:417:: with SMTP id 23mr16120588ywe.471.1543850447405; Mon, 03 Dec 2018 07:20:47 -0800 (PST) Received: from localhost ([107.15.81.208]) by smtp.gmail.com with ESMTPSA id u2sm5519915ywe.84.2018.12.03.07.20.45 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 03 Dec 2018 07:20:46 -0800 (PST) From: Josef Bacik To: linux-btrfs@vger.kernel.org, kernel-team@fb.com Cc: Josef Bacik Subject: [PATCH 03/10] btrfs: cleanup extent_op handling Date: Mon, 3 Dec 2018 10:20:31 -0500 Message-Id: <20181203152038.21388-4-josef@toxicpanda.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20181203152038.21388-1-josef@toxicpanda.com> References: <20181203152038.21388-1-josef@toxicpanda.com> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Josef Bacik The cleanup_extent_op function actually would run the extent_op if it needed running, which made the name sort of a misnomer. Change it to run_and_cleanup_extent_op, and move the actual cleanup work to cleanup_extent_op so it can be used by check_ref_cleanup() in order to unify the extent op handling. Reviewed-by: Lu Fengqi Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 35 ++++++++++++++++++++++------------- 1 file changed, 22 insertions(+), 13 deletions(-) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index e3ed3507018d..9f169f3c5fdb 100644 --- a/fs/btrfs/extent-tree.c +++ b/fs/btrfs/extent-tree.c @@ -2424,19 +2424,32 @@ static void unselect_delayed_ref_head(struct btrfs_delayed_ref_root *delayed_ref btrfs_delayed_ref_unlock(head); } -static int cleanup_extent_op(struct btrfs_trans_handle *trans, - struct btrfs_delayed_ref_head *head) +static struct btrfs_delayed_extent_op *cleanup_extent_op( + struct btrfs_delayed_ref_head *head) { struct btrfs_delayed_extent_op *extent_op = head->extent_op; - int ret; if (!extent_op) - return 0; - head->extent_op = NULL; + return NULL; + if (head->must_insert_reserved) { + head->extent_op = NULL; btrfs_free_delayed_extent_op(extent_op); - return 0; + return NULL; } + return extent_op; +} + +static int run_and_cleanup_extent_op(struct btrfs_trans_handle *trans, + struct btrfs_delayed_ref_head *head) +{ + struct btrfs_delayed_extent_op *extent_op; + int ret; + + extent_op = cleanup_extent_op(head); + if (!extent_op) + return 0; + head->extent_op = NULL; spin_unlock(&head->lock); ret = run_delayed_extent_op(trans, head, extent_op); btrfs_free_delayed_extent_op(extent_op); @@ -2488,7 +2501,7 @@ static int cleanup_ref_head(struct btrfs_trans_handle *trans, delayed_refs = &trans->transaction->delayed_refs; - ret = cleanup_extent_op(trans, head); + ret = run_and_cleanup_extent_op(trans, head); if (ret < 0) { unselect_delayed_ref_head(delayed_refs, head); btrfs_debug(fs_info, "run_delayed_extent_op returned %d", ret); @@ -6977,12 +6990,8 @@ static noinline int check_ref_cleanup(struct btrfs_trans_handle *trans, if (!RB_EMPTY_ROOT(&head->ref_tree.rb_root)) goto out; - if (head->extent_op) { - if (!head->must_insert_reserved) - goto out; - btrfs_free_delayed_extent_op(head->extent_op); - head->extent_op = NULL; - } + if (cleanup_extent_op(head) != NULL) + goto out; /* * waiting for the lock here would deadlock. If someone else has it From patchwork Mon Dec 3 15:20:32 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 10709803 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0D3BB17D5 for ; Mon, 3 Dec 2018 15:20:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F11932B25C for ; Mon, 3 Dec 2018 15:20:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E58772B437; Mon, 3 Dec 2018 15:20:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 67D1D2B3E3 for ; Mon, 3 Dec 2018 15:20:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726700AbeLCPUu (ORCPT ); Mon, 3 Dec 2018 10:20:50 -0500 Received: from mail-yb1-f195.google.com ([209.85.219.195]:35762 "EHLO mail-yb1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726549AbeLCPUu (ORCPT ); Mon, 3 Dec 2018 10:20:50 -0500 Received: by mail-yb1-f195.google.com with SMTP id z2-v6so5419302ybj.2 for ; Mon, 03 Dec 2018 07:20:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=kGdy1v9lj83bGenqQECPukc02qedRDiXFUktHQZr0jA=; b=W2cESaGMGPtij3yN9CS4udGTR7cRjSHs+oprgagbME+Qm5RckBAQVF5jSJxFDOnSmR RPc9CEHCqBSDyGbQrd+w18vXGvuiXEvwZKtFa2isduUrFRZy3VLMwdUfw2Ci6wf2v4Ja VxbYhIH7m+LX1DPeW6e/6Fmvxq7O8oTSoDiJ4hOk40OrHbXdYdcu850Kq4yR16FnNn6Q PFv4m1IR2MQ7yfNpPC7RF3B6bY5RTfhcJ45obfD/QtXKgAkuJkIPYDw19oOyv+EDIHG5 dtpuvMbwZ/XzbF8r3Wii7PFnCT3TK8I76XVaP7QkaOgyR9n3Kp1xVLR5zvC9hNpi1pDv j5UA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=kGdy1v9lj83bGenqQECPukc02qedRDiXFUktHQZr0jA=; b=uF6kd7h4Tz42I9ofHph+IcxPggucgHeDUErIVGq6crdbf47xllJGjQgk0rTlAAdnWn OO7KVR+sAfFlljZZLLOt64ouo4yin5PzxH1XC8oLoMf3gzq6qQuvctUvS4TFTODrrM9z pfS27uvOFkBoEh6AtxJ7b3bM6SA/GB2Ykr/KZgiU2qxWfv6VKx7kuC2VkXZZYt8ysJ/f lBaZDAlQs1pe50iknS3nNMFWvI8N2xTz3SVT7jiJxfIuBbb6JZfsFEJhpEMuwYFE8+nV bsG+IeJQSPKVR1rcECSVAHc1xvklCogmJstb5wDysOmSEL1/qeku8wOJ7PWD4i7/t7sC YnIw== X-Gm-Message-State: AA+aEWZERvBsZSYCe7xBy60FvVi1/qCQPqDQJGaov7mSU0lb7vYKg+H8 LX0qEAGsZZvnaYqganJYcp5+vt8NMTs= X-Google-Smtp-Source: AFSGD/Wv2ay1stsx4cqem/0bo+SHyROM7PAPNZ0HQ7kWFAl40GLiWP0KPTiHAH7zAgEXuZi7zqZUDQ== X-Received: by 2002:a25:9185:: with SMTP id w5-v6mr16177951ybl.138.1543850449073; Mon, 03 Dec 2018 07:20:49 -0800 (PST) Received: from localhost ([107.15.81.208]) by smtp.gmail.com with ESMTPSA id o14sm6010631ywo.52.2018.12.03.07.20.48 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 03 Dec 2018 07:20:48 -0800 (PST) From: Josef Bacik To: linux-btrfs@vger.kernel.org, kernel-team@fb.com Cc: Josef Bacik Subject: [PATCH 04/10] btrfs: only track ref_heads in delayed_ref_updates Date: Mon, 3 Dec 2018 10:20:32 -0500 Message-Id: <20181203152038.21388-5-josef@toxicpanda.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20181203152038.21388-1-josef@toxicpanda.com> References: <20181203152038.21388-1-josef@toxicpanda.com> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Josef Bacik We use this number to figure out how many delayed refs to run, but __btrfs_run_delayed_refs really only checks every time we need a new delayed ref head, so we always run at least one ref head completely no matter what the number of items on it. Fix the accounting to only be adjusted when we add/remove a ref head. Reviewed-by: Nikolay Borisov Signed-off-by: Josef Bacik --- fs/btrfs/delayed-ref.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/fs/btrfs/delayed-ref.c b/fs/btrfs/delayed-ref.c index b3e4c9fcb664..48725fa757a3 100644 --- a/fs/btrfs/delayed-ref.c +++ b/fs/btrfs/delayed-ref.c @@ -251,8 +251,6 @@ static inline void drop_delayed_ref(struct btrfs_trans_handle *trans, ref->in_tree = 0; btrfs_put_delayed_ref(ref); atomic_dec(&delayed_refs->num_entries); - if (trans->delayed_ref_updates) - trans->delayed_ref_updates--; } static bool merge_ref(struct btrfs_trans_handle *trans, @@ -467,7 +465,6 @@ static int insert_delayed_ref(struct btrfs_trans_handle *trans, if (ref->action == BTRFS_ADD_DELAYED_REF) list_add_tail(&ref->add_list, &href->ref_add_list); atomic_inc(&root->num_entries); - trans->delayed_ref_updates++; spin_unlock(&href->lock); return ret; } From patchwork Mon Dec 3 15:20:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 10709807 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1ADD518BC for ; Mon, 3 Dec 2018 15:20:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0441A2AF5B for ; Mon, 3 Dec 2018 15:20:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 01E442B252; Mon, 3 Dec 2018 15:20:55 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 79B4F2B3C7 for ; Mon, 3 Dec 2018 15:20:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726706AbeLCPUx (ORCPT ); Mon, 3 Dec 2018 10:20:53 -0500 Received: from mail-yw1-f68.google.com ([209.85.161.68]:38874 "EHLO mail-yw1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726549AbeLCPUx (ORCPT ); Mon, 3 Dec 2018 10:20:53 -0500 Received: by mail-yw1-f68.google.com with SMTP id i20so5522532ywc.5 for ; Mon, 03 Dec 2018 07:20:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=L6oGBPEzscGKsaf2nJZ8ggM/w/zhzdZHJKPMzya7Ipk=; b=TlTjMfLZw8sH+Fylsd552p2Ese7Dyro/fAOFGXXviGmeiDBlZByjZOrKkxfcqguu3L Pfv7LWcOuPpxfMZg/G4eOTIVD3RexVeCLwJibLWSs97YNmKpgHupfE5PIdGuUxzM/AJC sMbFWtCZOVWo+x+JQXSPGU7ZWghRqWrVOfhlZbe9Jw/JADrbKABhehd7QSCp7bFGjCcx T4okAyem19Y5DsBp3MPFXd9qPc1JOxRAjdPcf5pu2hn77/eRupjldFYHkvrOj8UuhyMu 7Wc6oReyFk8z6TBnr6dV+qT2CDjh6HMKffkJSCzSF3zYi/lXEAII+M9NXTInI0YoQYvj oSWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=L6oGBPEzscGKsaf2nJZ8ggM/w/zhzdZHJKPMzya7Ipk=; b=gSYpjXBdnc384ja/Sny7zNQznzmYvdC7kEk9voPbvFWaufax99u3Xxn7DGrpjKSSR8 qeYvHuj/ZRg+PTcgBrgUM1sbFm1UNfHJ6odJXdRWFT3wSmNZLUspCtixUJGT4mVksvsI 0+gomMJFgQB97WH4xieDtnhfT/7L6WApR9PxqeeSUBA2vLUODgEqd9Im5fTNUEtNicIq u77gQbAllLAqcW42gC0/oeMKlSzTBehcuK7DHQe2w33WZi+rqRkimY5SN8DYNaRy6bwZ eOWvUJQEcgKlW4PoYK3SzI1Qi6/RuP47/6WBCykAT/YNLj5hHAn8Xz5oPh5AMt5RpVfp 3t2Q== X-Gm-Message-State: AA+aEWZKOpgZtyZL4Zv8tTytw5JlpDZFSv9+BzRE2NkxF6cpV+hOWzTq EzHZ3BPQHPk6nawzzoDwkwZMItomRW4= X-Google-Smtp-Source: AFSGD/V/S+FTeexZpFjMxSy3uMMGEQkrfJ5UiBcVIef1+SSoA+N1m4bOT6L7bOVjtFctZpeSZw9VEA== X-Received: by 2002:a81:7c04:: with SMTP id x4mr7294527ywc.264.1543850450854; Mon, 03 Dec 2018 07:20:50 -0800 (PST) Received: from localhost ([107.15.81.208]) by smtp.gmail.com with ESMTPSA id w2sm11780259ywl.10.2018.12.03.07.20.49 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 03 Dec 2018 07:20:50 -0800 (PST) From: Josef Bacik To: linux-btrfs@vger.kernel.org, kernel-team@fb.com Cc: Josef Bacik Subject: [PATCH 05/10] btrfs: introduce delayed_refs_rsv Date: Mon, 3 Dec 2018 10:20:33 -0500 Message-Id: <20181203152038.21388-6-josef@toxicpanda.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20181203152038.21388-1-josef@toxicpanda.com> References: <20181203152038.21388-1-josef@toxicpanda.com> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Josef Bacik Traditionally we've had voodoo in btrfs to account for the space that delayed refs may take up by having a global_block_rsv. This works most of the time, except when it doesn't. We've had issues reported and seen in production where sometimes the global reserve is exhausted during transaction commit before we can run all of our delayed refs, resulting in an aborted transaction. Because of this voodoo we have equally dubious flushing semantics around throttling delayed refs which we often get wrong. So instead give them their own block_rsv. This way we can always know exactly how much outstanding space we need for delayed refs. This allows us to make sure we are constantly filling that reservation up with space, and allows us to put more precise pressure on the enospc system. Instead of doing math to see if its a good time to throttle, the normal enospc code will be invoked if we have a lot of delayed refs pending, and they will be run via the normal flushing mechanism. For now the delayed_refs_rsv will hold the reservations for the delayed refs, the block group updates, and deleting csums. We could have a separate rsv for the block group updates, but the csum deletion stuff is still handled via the delayed_refs so that will stay there. Signed-off-by: Josef Bacik --- fs/btrfs/ctree.h | 14 +++- fs/btrfs/delayed-ref.c | 43 ++++++++-- fs/btrfs/disk-io.c | 4 + fs/btrfs/extent-tree.c | 212 +++++++++++++++++++++++++++++++++++++++++++++---- fs/btrfs/transaction.c | 37 ++++++++- 5 files changed, 284 insertions(+), 26 deletions(-) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h index 8b41ec42f405..52a87d446945 100644 --- a/fs/btrfs/ctree.h +++ b/fs/btrfs/ctree.h @@ -448,8 +448,9 @@ struct btrfs_space_info { #define BTRFS_BLOCK_RSV_TRANS 3 #define BTRFS_BLOCK_RSV_CHUNK 4 #define BTRFS_BLOCK_RSV_DELOPS 5 -#define BTRFS_BLOCK_RSV_EMPTY 6 -#define BTRFS_BLOCK_RSV_TEMP 7 +#define BTRFS_BLOCK_RSV_DELREFS 6 +#define BTRFS_BLOCK_RSV_EMPTY 7 +#define BTRFS_BLOCK_RSV_TEMP 8 struct btrfs_block_rsv { u64 size; @@ -812,6 +813,8 @@ struct btrfs_fs_info { struct btrfs_block_rsv chunk_block_rsv; /* block reservation for delayed operations */ struct btrfs_block_rsv delayed_block_rsv; + /* block reservation for delayed refs */ + struct btrfs_block_rsv delayed_refs_rsv; struct btrfs_block_rsv empty_block_rsv; @@ -2796,6 +2799,13 @@ int btrfs_cond_migrate_bytes(struct btrfs_fs_info *fs_info, void btrfs_block_rsv_release(struct btrfs_fs_info *fs_info, struct btrfs_block_rsv *block_rsv, u64 num_bytes); +void btrfs_delayed_refs_rsv_release(struct btrfs_fs_info *fs_info, int nr); +void btrfs_update_delayed_refs_rsv(struct btrfs_trans_handle *trans); +int btrfs_delayed_refs_rsv_refill(struct btrfs_fs_info *fs_info, + enum btrfs_reserve_flush_enum flush); +void btrfs_migrate_to_delayed_refs_rsv(struct btrfs_fs_info *fs_info, + struct btrfs_block_rsv *src, + u64 num_bytes); int btrfs_inc_block_group_ro(struct btrfs_block_group_cache *cache); void btrfs_dec_block_group_ro(struct btrfs_block_group_cache *cache); void btrfs_put_block_group_cache(struct btrfs_fs_info *info); diff --git a/fs/btrfs/delayed-ref.c b/fs/btrfs/delayed-ref.c index 48725fa757a3..a198ab91879c 100644 --- a/fs/btrfs/delayed-ref.c +++ b/fs/btrfs/delayed-ref.c @@ -474,11 +474,14 @@ static int insert_delayed_ref(struct btrfs_trans_handle *trans, * existing and update must have the same bytenr */ static noinline void -update_existing_head_ref(struct btrfs_delayed_ref_root *delayed_refs, +update_existing_head_ref(struct btrfs_trans_handle *trans, struct btrfs_delayed_ref_head *existing, struct btrfs_delayed_ref_head *update, int *old_ref_mod_ret) { + struct btrfs_delayed_ref_root *delayed_refs = + &trans->transaction->delayed_refs; + struct btrfs_fs_info *fs_info = trans->fs_info; int old_ref_mod; BUG_ON(existing->is_data != update->is_data); @@ -536,10 +539,18 @@ update_existing_head_ref(struct btrfs_delayed_ref_root *delayed_refs, * versa we need to make sure to adjust pending_csums accordingly. */ if (existing->is_data) { - if (existing->total_ref_mod >= 0 && old_ref_mod < 0) + u64 csum_leaves = + btrfs_csum_bytes_to_leaves(fs_info, + existing->num_bytes); + + if (existing->total_ref_mod >= 0 && old_ref_mod < 0) { delayed_refs->pending_csums -= existing->num_bytes; - if (existing->total_ref_mod < 0 && old_ref_mod >= 0) + btrfs_delayed_refs_rsv_release(fs_info, csum_leaves); + } + if (existing->total_ref_mod < 0 && old_ref_mod >= 0) { delayed_refs->pending_csums += existing->num_bytes; + trans->delayed_ref_updates += csum_leaves; + } } spin_unlock(&existing->lock); } @@ -645,7 +656,7 @@ add_delayed_ref_head(struct btrfs_trans_handle *trans, && head_ref->qgroup_reserved && existing->qgroup_ref_root && existing->qgroup_reserved); - update_existing_head_ref(delayed_refs, existing, head_ref, + update_existing_head_ref(trans, existing, head_ref, old_ref_mod); /* * we've updated the existing ref, free the newly @@ -656,8 +667,12 @@ add_delayed_ref_head(struct btrfs_trans_handle *trans, } else { if (old_ref_mod) *old_ref_mod = 0; - if (head_ref->is_data && head_ref->ref_mod < 0) + if (head_ref->is_data && head_ref->ref_mod < 0) { delayed_refs->pending_csums += head_ref->num_bytes; + trans->delayed_ref_updates += + btrfs_csum_bytes_to_leaves(trans->fs_info, + head_ref->num_bytes); + } delayed_refs->num_heads++; delayed_refs->num_heads_ready++; atomic_inc(&delayed_refs->num_entries); @@ -793,6 +808,12 @@ int btrfs_add_delayed_tree_ref(struct btrfs_trans_handle *trans, ret = insert_delayed_ref(trans, delayed_refs, head_ref, &ref->node); spin_unlock(&delayed_refs->lock); + /* + * Need to update the delayed_refs_rsv with any changes we may have + * made. + */ + btrfs_update_delayed_refs_rsv(trans); + trace_add_delayed_tree_ref(fs_info, &ref->node, ref, action == BTRFS_ADD_DELAYED_EXTENT ? BTRFS_ADD_DELAYED_REF : action); @@ -874,6 +895,12 @@ int btrfs_add_delayed_data_ref(struct btrfs_trans_handle *trans, ret = insert_delayed_ref(trans, delayed_refs, head_ref, &ref->node); spin_unlock(&delayed_refs->lock); + /* + * Need to update the delayed_refs_rsv with any changes we may have + * made. + */ + btrfs_update_delayed_refs_rsv(trans); + trace_add_delayed_data_ref(trans->fs_info, &ref->node, ref, action == BTRFS_ADD_DELAYED_EXTENT ? BTRFS_ADD_DELAYED_REF : action); @@ -910,6 +937,12 @@ int btrfs_add_delayed_extent_op(struct btrfs_fs_info *fs_info, NULL, NULL, NULL); spin_unlock(&delayed_refs->lock); + + /* + * Need to update the delayed_refs_rsv with any changes we may have + * made. + */ + btrfs_update_delayed_refs_rsv(trans); return 0; } diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c index 55e8ca782b98..beaa58e742b5 100644 --- a/fs/btrfs/disk-io.c +++ b/fs/btrfs/disk-io.c @@ -2666,6 +2666,9 @@ int open_ctree(struct super_block *sb, btrfs_init_block_rsv(&fs_info->empty_block_rsv, BTRFS_BLOCK_RSV_EMPTY); btrfs_init_block_rsv(&fs_info->delayed_block_rsv, BTRFS_BLOCK_RSV_DELOPS); + btrfs_init_block_rsv(&fs_info->delayed_refs_rsv, + BTRFS_BLOCK_RSV_DELREFS); + atomic_set(&fs_info->async_delalloc_pages, 0); atomic_set(&fs_info->defrag_running, 0); atomic_set(&fs_info->qgroup_op_seq, 0); @@ -4413,6 +4416,7 @@ void btrfs_cleanup_dirty_bgs(struct btrfs_transaction *cur_trans, spin_unlock(&cur_trans->dirty_bgs_lock); btrfs_put_block_group(cache); + btrfs_delayed_refs_rsv_release(fs_info, 1); spin_lock(&cur_trans->dirty_bgs_lock); } spin_unlock(&cur_trans->dirty_bgs_lock); diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index 9f169f3c5fdb..aa0a638d0263 100644 --- a/fs/btrfs/extent-tree.c +++ b/fs/btrfs/extent-tree.c @@ -2462,6 +2462,7 @@ static void cleanup_ref_head_accounting(struct btrfs_trans_handle *trans, struct btrfs_fs_info *fs_info = trans->fs_info; struct btrfs_delayed_ref_root *delayed_refs = &trans->transaction->delayed_refs; + int nr_items = 1; /* Dropping this ref head update. */ if (head->total_ref_mod < 0) { struct btrfs_space_info *space_info; @@ -2479,16 +2480,24 @@ static void cleanup_ref_head_accounting(struct btrfs_trans_handle *trans, -head->num_bytes, BTRFS_TOTAL_BYTES_PINNED_BATCH); + /* + * We had csum deletions accounted for in our delayed refs rsv, + * we need to drop the csum leaves for this update from our + * delayed_refs_rsv. + */ if (head->is_data) { spin_lock(&delayed_refs->lock); delayed_refs->pending_csums -= head->num_bytes; spin_unlock(&delayed_refs->lock); + nr_items += btrfs_csum_bytes_to_leaves(fs_info, + head->num_bytes); } } /* Also free its reserved qgroup space */ btrfs_qgroup_free_delayed_ref(fs_info, head->qgroup_ref_root, head->qgroup_reserved); + btrfs_delayed_refs_rsv_release(fs_info, nr_items); } static int cleanup_ref_head(struct btrfs_trans_handle *trans, @@ -3626,6 +3635,8 @@ int btrfs_start_dirty_block_groups(struct btrfs_trans_handle *trans) */ mutex_lock(&trans->transaction->cache_write_mutex); while (!list_empty(&dirty)) { + bool drop_reserve = true; + cache = list_first_entry(&dirty, struct btrfs_block_group_cache, dirty_list); @@ -3698,6 +3709,7 @@ int btrfs_start_dirty_block_groups(struct btrfs_trans_handle *trans) list_add_tail(&cache->dirty_list, &cur_trans->dirty_bgs); btrfs_get_block_group(cache); + drop_reserve = false; } spin_unlock(&cur_trans->dirty_bgs_lock); } else if (ret) { @@ -3708,6 +3720,8 @@ int btrfs_start_dirty_block_groups(struct btrfs_trans_handle *trans) /* if its not on the io list, we need to put the block group */ if (should_put) btrfs_put_block_group(cache); + if (drop_reserve) + btrfs_delayed_refs_rsv_release(fs_info, 1); if (ret) break; @@ -3856,6 +3870,7 @@ int btrfs_write_dirty_block_groups(struct btrfs_trans_handle *trans, /* if its not on the io list, we need to put the block group */ if (should_put) btrfs_put_block_group(cache); + btrfs_delayed_refs_rsv_release(fs_info, 1); spin_lock(&cur_trans->dirty_bgs_lock); } spin_unlock(&cur_trans->dirty_bgs_lock); @@ -5389,6 +5404,91 @@ int btrfs_cond_migrate_bytes(struct btrfs_fs_info *fs_info, return 0; } +/** + * btrfs_migrate_to_delayed_refs_rsv - transfer bytes to our delayed refs rsv. + * @fs_info - the fs info for our fs. + * @src - the source block rsv to transfer from. + * @num_bytes - the number of bytes to transfer. + * + * This transfers up to the num_bytes amount from the src rsv to the + * delayed_refs_rsv. Any extra bytes are returned to the space info. + */ +void btrfs_migrate_to_delayed_refs_rsv(struct btrfs_fs_info *fs_info, + struct btrfs_block_rsv *src, + u64 num_bytes) +{ + struct btrfs_block_rsv *delayed_refs_rsv = &fs_info->delayed_refs_rsv; + u64 to_free = 0; + + spin_lock(&src->lock); + src->reserved -= num_bytes; + src->size -= num_bytes; + spin_unlock(&src->lock); + + spin_lock(&delayed_refs_rsv->lock); + if (delayed_refs_rsv->size > delayed_refs_rsv->reserved) { + u64 delta = delayed_refs_rsv->size - + delayed_refs_rsv->reserved; + if (num_bytes > delta) { + to_free = num_bytes - delta; + num_bytes = delta; + } + } else { + to_free = num_bytes; + num_bytes = 0; + } + + if (num_bytes) + delayed_refs_rsv->reserved += num_bytes; + if (delayed_refs_rsv->reserved >= delayed_refs_rsv->size) + delayed_refs_rsv->full = 1; + spin_unlock(&delayed_refs_rsv->lock); + + if (num_bytes) + trace_btrfs_space_reservation(fs_info, "delayed_refs_rsv", + 0, num_bytes, 1); + if (to_free) + space_info_add_old_bytes(fs_info, delayed_refs_rsv->space_info, + to_free); +} + +/** + * btrfs_delayed_refs_rsv_refill - refill based on our delayed refs usage. + * @fs_info - the fs_info for our fs. + * @flush - control how we can flush for this reservation. + * + * This will refill the delayed block_rsv up to 1 items size worth of space and + * will return -ENOSPC if we can't make the reservation. + */ +int btrfs_delayed_refs_rsv_refill(struct btrfs_fs_info *fs_info, + enum btrfs_reserve_flush_enum flush) +{ + struct btrfs_block_rsv *block_rsv = &fs_info->delayed_refs_rsv; + u64 limit = btrfs_calc_trans_metadata_size(fs_info, 1); + u64 num_bytes = 0; + int ret = -ENOSPC; + + spin_lock(&block_rsv->lock); + if (block_rsv->reserved < block_rsv->size) { + num_bytes = block_rsv->size - block_rsv->reserved; + num_bytes = min(num_bytes, limit); + } + spin_unlock(&block_rsv->lock); + + if (!num_bytes) + return 0; + + ret = reserve_metadata_bytes(fs_info->extent_root, block_rsv, + num_bytes, flush); + if (ret) + return ret; + block_rsv_add_bytes(block_rsv, num_bytes, 0); + trace_btrfs_space_reservation(fs_info, "delayed_refs_rsv", + 0, num_bytes, 1); + return 0; +} + + /* * This is for space we already have accounted in space_info->bytes_may_use, so * basically when we're returning space from block_rsv's. @@ -5709,6 +5809,31 @@ static int btrfs_inode_rsv_refill(struct btrfs_inode *inode, return ret; } +static u64 __btrfs_block_rsv_release(struct btrfs_fs_info *fs_info, + struct btrfs_block_rsv *block_rsv, + u64 num_bytes, u64 *qgroup_to_release) +{ + struct btrfs_block_rsv *global_rsv = &fs_info->global_block_rsv; + struct btrfs_block_rsv *delayed_rsv = &fs_info->delayed_refs_rsv; + struct btrfs_block_rsv *target = delayed_rsv; + + if (target->full || target == block_rsv) + target = global_rsv; + + if (block_rsv->space_info != target->space_info) + target = NULL; + + return block_rsv_release_bytes(fs_info, block_rsv, target, num_bytes, + qgroup_to_release); +} + +void btrfs_block_rsv_release(struct btrfs_fs_info *fs_info, + struct btrfs_block_rsv *block_rsv, + u64 num_bytes) +{ + __btrfs_block_rsv_release(fs_info, block_rsv, num_bytes, NULL); +} + /** * btrfs_inode_rsv_release - release any excessive reservation. * @inode - the inode we need to release from. @@ -5723,7 +5848,6 @@ static int btrfs_inode_rsv_refill(struct btrfs_inode *inode, static void btrfs_inode_rsv_release(struct btrfs_inode *inode, bool qgroup_free) { struct btrfs_fs_info *fs_info = inode->root->fs_info; - struct btrfs_block_rsv *global_rsv = &fs_info->global_block_rsv; struct btrfs_block_rsv *block_rsv = &inode->block_rsv; u64 released = 0; u64 qgroup_to_release = 0; @@ -5733,8 +5857,8 @@ static void btrfs_inode_rsv_release(struct btrfs_inode *inode, bool qgroup_free) * are releasing 0 bytes, and then we'll just get the reservation over * the size free'd. */ - released = block_rsv_release_bytes(fs_info, block_rsv, global_rsv, 0, - &qgroup_to_release); + released = __btrfs_block_rsv_release(fs_info, block_rsv, 0, + &qgroup_to_release); if (released > 0) trace_btrfs_space_reservation(fs_info, "delalloc", btrfs_ino(inode), released, 0); @@ -5745,16 +5869,26 @@ static void btrfs_inode_rsv_release(struct btrfs_inode *inode, bool qgroup_free) qgroup_to_release); } -void btrfs_block_rsv_release(struct btrfs_fs_info *fs_info, - struct btrfs_block_rsv *block_rsv, - u64 num_bytes) +/** + * btrfs_delayed_refs_rsv_release - release a ref head's reservation. + * @fs_info - the fs_info for our fs. + * @nr - the number of items to drop. + * + * This drops the delayed ref head's count from the delayed refs rsv and free's + * any excess reservation we had. + */ +void btrfs_delayed_refs_rsv_release(struct btrfs_fs_info *fs_info, int nr) { + struct btrfs_block_rsv *block_rsv = &fs_info->delayed_refs_rsv; struct btrfs_block_rsv *global_rsv = &fs_info->global_block_rsv; + u64 num_bytes = btrfs_calc_trans_metadata_size(fs_info, nr); + u64 released = 0; - if (global_rsv == block_rsv || - block_rsv->space_info != global_rsv->space_info) - global_rsv = NULL; - block_rsv_release_bytes(fs_info, block_rsv, global_rsv, num_bytes, NULL); + released = block_rsv_release_bytes(fs_info, block_rsv, global_rsv, + num_bytes, NULL); + if (released) + trace_btrfs_space_reservation(fs_info, "delayed_refs_rsv", + 0, released, 0); } static void update_global_block_rsv(struct btrfs_fs_info *fs_info) @@ -5819,9 +5953,10 @@ static void init_global_block_rsv(struct btrfs_fs_info *fs_info) fs_info->trans_block_rsv.space_info = space_info; fs_info->empty_block_rsv.space_info = space_info; fs_info->delayed_block_rsv.space_info = space_info; + fs_info->delayed_refs_rsv.space_info = space_info; - fs_info->extent_root->block_rsv = &fs_info->global_block_rsv; - fs_info->csum_root->block_rsv = &fs_info->global_block_rsv; + fs_info->extent_root->block_rsv = &fs_info->delayed_refs_rsv; + fs_info->csum_root->block_rsv = &fs_info->delayed_refs_rsv; fs_info->dev_root->block_rsv = &fs_info->global_block_rsv; fs_info->tree_root->block_rsv = &fs_info->global_block_rsv; if (fs_info->quota_root) @@ -5841,8 +5976,34 @@ static void release_global_block_rsv(struct btrfs_fs_info *fs_info) WARN_ON(fs_info->chunk_block_rsv.reserved > 0); WARN_ON(fs_info->delayed_block_rsv.size > 0); WARN_ON(fs_info->delayed_block_rsv.reserved > 0); + WARN_ON(fs_info->delayed_refs_rsv.reserved > 0); + WARN_ON(fs_info->delayed_refs_rsv.size > 0); } +/* + * btrfs_update_delayed_refs_rsv - adjust the size of the delayed refs rsv + * @trans - the trans that may have generated delayed refs + * + * This is to be called anytime we may have adjusted trans->delayed_ref_updates, + * it'll calculate the additional size and add it to the delayed_refs_rsv. + */ +void btrfs_update_delayed_refs_rsv(struct btrfs_trans_handle *trans) +{ + struct btrfs_fs_info *fs_info = trans->fs_info; + struct btrfs_block_rsv *delayed_rsv = &fs_info->delayed_refs_rsv; + u64 num_bytes; + + if (!trans->delayed_ref_updates) + return; + + num_bytes = btrfs_calc_trans_metadata_size(fs_info, + trans->delayed_ref_updates); + spin_lock(&delayed_rsv->lock); + delayed_rsv->size += num_bytes; + delayed_rsv->full = 0; + spin_unlock(&delayed_rsv->lock); + trans->delayed_ref_updates = 0; +} /* * To be called after all the new block groups attached to the transaction @@ -6135,6 +6296,7 @@ static int update_block_group(struct btrfs_trans_handle *trans, u64 old_val; u64 byte_in_group; int factor; + int ret = 0; /* block accounting for super block */ spin_lock(&info->delalloc_root_lock); @@ -6148,8 +6310,10 @@ static int update_block_group(struct btrfs_trans_handle *trans, while (total) { cache = btrfs_lookup_block_group(info, bytenr); - if (!cache) - return -ENOENT; + if (!cache) { + ret = -ENOENT; + break; + } factor = btrfs_bg_type_to_factor(cache->flags); /* @@ -6208,6 +6372,7 @@ static int update_block_group(struct btrfs_trans_handle *trans, list_add_tail(&cache->dirty_list, &trans->transaction->dirty_bgs); trans->transaction->num_dirty_bgs++; + trans->delayed_ref_updates++; btrfs_get_block_group(cache); } spin_unlock(&trans->transaction->dirty_bgs_lock); @@ -6225,7 +6390,10 @@ static int update_block_group(struct btrfs_trans_handle *trans, total -= num_bytes; bytenr += num_bytes; } - return 0; + + /* Modified block groups are accounted for in the delayed_refs_rsv. */ + btrfs_update_delayed_refs_rsv(trans); + return ret; } static u64 first_logical_byte(struct btrfs_fs_info *fs_info, u64 search_start) @@ -8368,7 +8536,12 @@ use_block_rsv(struct btrfs_trans_handle *trans, goto again; } - if (btrfs_test_opt(fs_info, ENOSPC_DEBUG)) { + /* + * The global reserve still exists to save us from ourselves, so don't + * warn_on if we are short on our delayed refs reserve. + */ + if (block_rsv->type != BTRFS_BLOCK_RSV_DELREFS && + btrfs_test_opt(fs_info, ENOSPC_DEBUG)) { static DEFINE_RATELIMIT_STATE(_rs, DEFAULT_RATELIMIT_INTERVAL * 10, /*DEFAULT_RATELIMIT_BURST*/ 1); @@ -10301,6 +10474,7 @@ void btrfs_create_pending_block_groups(struct btrfs_trans_handle *trans) add_block_group_free_space(trans, block_group); /* already aborted the transaction if it failed. */ next: + btrfs_delayed_refs_rsv_release(fs_info, 1); list_del_init(&block_group->bg_list); } btrfs_trans_release_chunk_metadata(trans); @@ -10378,6 +10552,8 @@ int btrfs_make_block_group(struct btrfs_trans_handle *trans, u64 bytes_used, link_block_group(cache); list_add_tail(&cache->bg_list, &trans->new_bgs); + trans->delayed_ref_updates++; + btrfs_update_delayed_refs_rsv(trans); set_avail_alloc_bits(fs_info, type); return 0; @@ -10415,6 +10591,7 @@ int btrfs_remove_block_group(struct btrfs_trans_handle *trans, int factor; struct btrfs_caching_control *caching_ctl = NULL; bool remove_em; + bool remove_rsv = false; block_group = btrfs_lookup_block_group(fs_info, group_start); BUG_ON(!block_group); @@ -10479,6 +10656,7 @@ int btrfs_remove_block_group(struct btrfs_trans_handle *trans, if (!list_empty(&block_group->dirty_list)) { list_del_init(&block_group->dirty_list); + remove_rsv = true; btrfs_put_block_group(block_group); } spin_unlock(&trans->transaction->dirty_bgs_lock); @@ -10688,6 +10866,8 @@ int btrfs_remove_block_group(struct btrfs_trans_handle *trans, ret = btrfs_del_item(trans, root, path); out: + if (remove_rsv) + btrfs_delayed_refs_rsv_release(fs_info, 1); btrfs_free_path(path); return ret; } diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c index f92c0a88c4ad..a21c4defad92 100644 --- a/fs/btrfs/transaction.c +++ b/fs/btrfs/transaction.c @@ -454,7 +454,7 @@ start_transaction(struct btrfs_root *root, unsigned int num_items, bool enforce_qgroups) { struct btrfs_fs_info *fs_info = root->fs_info; - + struct btrfs_block_rsv *delayed_refs_rsv = &fs_info->delayed_refs_rsv; struct btrfs_trans_handle *h; struct btrfs_transaction *cur_trans; u64 num_bytes = 0; @@ -483,13 +483,28 @@ start_transaction(struct btrfs_root *root, unsigned int num_items, * the appropriate flushing if need be. */ if (num_items && root != fs_info->chunk_root) { + struct btrfs_block_rsv *rsv = &fs_info->trans_block_rsv; + u64 delayed_refs_bytes = 0; + qgroup_reserved = num_items * fs_info->nodesize; ret = btrfs_qgroup_reserve_meta_pertrans(root, qgroup_reserved, enforce_qgroups); if (ret) return ERR_PTR(ret); + /* + * We want to reserve all the bytes we may need all at once, so + * we only do 1 enospc flushing cycle per transaction start. We + * accomplish this by simply assuming we'll do 2 x num_items + * worth of delayed refs updates in this trans handle, and + * refill that amount for whatever is missing in the reserve. + */ num_bytes = btrfs_calc_trans_metadata_size(fs_info, num_items); + if (delayed_refs_rsv->full == 0) { + delayed_refs_bytes = num_bytes; + num_bytes <<= 1; + } + /* * Do the reservation for the relocation root creation */ @@ -498,8 +513,24 @@ start_transaction(struct btrfs_root *root, unsigned int num_items, reloc_reserved = true; } - ret = btrfs_block_rsv_add(root, &fs_info->trans_block_rsv, - num_bytes, flush); + ret = btrfs_block_rsv_add(root, rsv, num_bytes, flush); + if (ret) + goto reserve_fail; + if (delayed_refs_bytes) { + btrfs_migrate_to_delayed_refs_rsv(fs_info, rsv, + delayed_refs_bytes); + num_bytes -= delayed_refs_bytes; + } + } else if (num_items == 0 && flush == BTRFS_RESERVE_FLUSH_ALL && + !delayed_refs_rsv->full) { + /* + * Some people call with btrfs_start_transaction(root, 0) + * because they can be throttled, but have some other mechanism + * for reserving space. We still want these guys to refill the + * delayed block_rsv so just add 1 items worth of reservation + * here. + */ + ret = btrfs_delayed_refs_rsv_refill(fs_info, flush); if (ret) goto reserve_fail; } From patchwork Mon Dec 3 15:20:34 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 10709805 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EF61514E2 for ; Mon, 3 Dec 2018 15:20:55 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DA8952B25F for ; Mon, 3 Dec 2018 15:20:55 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CEF482B252; Mon, 3 Dec 2018 15:20:55 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E1AD92B25F for ; Mon, 3 Dec 2018 15:20:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726708AbeLCPUy (ORCPT ); Mon, 3 Dec 2018 10:20:54 -0500 Received: from mail-yw1-f68.google.com ([209.85.161.68]:43058 "EHLO mail-yw1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726701AbeLCPUy (ORCPT ); Mon, 3 Dec 2018 10:20:54 -0500 Received: by mail-yw1-f68.google.com with SMTP id l200so5505629ywe.10 for ; Mon, 03 Dec 2018 07:20:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references; bh=nJ9I87mj/kf+AeJTopTyg8uLem/EFFwXtsk1rurU9IE=; b=iX8THiQP5MV7+42h31xn5CenyfOhiBlR5lxNTro5roWNVQINQnLrV8yn+0Rh26oT1G uYs282vLPZW3NEsDbVv0CMcB9dCjOq1V9zGF7YHcKIzFyl03EQuXheVT3LeIrouHjMyS QdTcUvzRCG6TKtgHeGrmIq/RAwNormjMXi+RaTzTKVZ27RpI6oDeRdqMqaUcsf7njqse 84xu6j2bCb5LVyiNiEoqY95VEB+95LfP9pjcVjUld9f8+ZeSpBCpq6rGq2jj8oMXNLRb ZwQi4tzFn1nTRl45jWRA0je6Dm4Jrcx58eqy/l3s3gviCfhH1Dl79ilJgHthXbVAS/rl ubrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=nJ9I87mj/kf+AeJTopTyg8uLem/EFFwXtsk1rurU9IE=; b=F/GS+q8gfgBAzEqRnVXGwZr9+l7L+qcB+q1oeWgDwZF6AJoGllz3fA2/p7C5dTlX7J R30gshMnOAXX0gxNvjgng7W8qagRumwULw1psYc5RGXg+xkvjKbbdA05sgfJQQHnjcgu wJ+l9+q0OHNxFDX/FYmRA+YgofifmBA2WK3YETHikAzylvi1c1cOeVFqU7wYE4wk4LWm CroqU3CmhJNPaWBBTO/S70pRKntXRJOO7T276YDOxRpyKrxKKft2RTr3BKoxKokGzmrA yNFEipcJ3qefgLrh7+aXJPeQfFptw5joze5cyzk+vJtC7eRkICiepaZR8IpFHiCEBSVr N+iQ== X-Gm-Message-State: AA+aEWYj2QVLl3YYLigknNATUvXTxGNeYFo8ylEPPDB5x9BOwjMh8QaC lSQEc5bD/AoBWSYDaXqUoWLuamdHDEQ= X-Google-Smtp-Source: AFSGD/W5GCBK2VfH1lStrvwRm98OkiPqyRNTXdQpJkoqks8xooIkYY0VaEAJLcwNsZ3HwSTwuUTPwg== X-Received: by 2002:a81:6c51:: with SMTP id h78mr16045960ywc.116.1543850452494; Mon, 03 Dec 2018 07:20:52 -0800 (PST) Received: from localhost ([107.15.81.208]) by smtp.gmail.com with ESMTPSA id u2sm5520251ywe.84.2018.12.03.07.20.51 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 03 Dec 2018 07:20:51 -0800 (PST) From: Josef Bacik To: linux-btrfs@vger.kernel.org, kernel-team@fb.com Subject: [PATCH 06/10] btrfs: update may_commit_transaction to use the delayed refs rsv Date: Mon, 3 Dec 2018 10:20:34 -0500 Message-Id: <20181203152038.21388-7-josef@toxicpanda.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20181203152038.21388-1-josef@toxicpanda.com> References: <20181203152038.21388-1-josef@toxicpanda.com> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Any space used in the delayed_refs_rsv will be freed up by a transaction commit, so instead of just counting the pinned space we also need to account for any space in the delayed_refs_rsv when deciding if it will make a different to commit the transaction to satisfy our space reservation. If we have enough bytes to satisfy our reservation ticket then we are good to go, otherwise subtract out what space we would gain back by committing the transaction and compare that against the pinned space to make our decision. Signed-off-by: Josef Bacik Reviewed-by: Nikolay Borisov --- fs/btrfs/extent-tree.c | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index aa0a638d0263..63ff9d832867 100644 --- a/fs/btrfs/extent-tree.c +++ b/fs/btrfs/extent-tree.c @@ -4843,8 +4843,10 @@ static int may_commit_transaction(struct btrfs_fs_info *fs_info, { struct reserve_ticket *ticket = NULL; struct btrfs_block_rsv *delayed_rsv = &fs_info->delayed_block_rsv; + struct btrfs_block_rsv *delayed_refs_rsv = &fs_info->delayed_refs_rsv; struct btrfs_trans_handle *trans; - u64 bytes; + u64 bytes_needed; + u64 reclaim_bytes = 0; trans = (struct btrfs_trans_handle *)current->journal_info; if (trans) @@ -4857,15 +4859,15 @@ static int may_commit_transaction(struct btrfs_fs_info *fs_info, else if (!list_empty(&space_info->tickets)) ticket = list_first_entry(&space_info->tickets, struct reserve_ticket, list); - bytes = (ticket) ? ticket->bytes : 0; + bytes_needed = (ticket) ? ticket->bytes : 0; spin_unlock(&space_info->lock); - if (!bytes) + if (!bytes_needed) return 0; /* See if there is enough pinned space to make this reservation */ if (__percpu_counter_compare(&space_info->total_bytes_pinned, - bytes, + bytes_needed, BTRFS_TOTAL_BYTES_PINNED_BATCH) >= 0) goto commit; @@ -4877,14 +4879,18 @@ static int may_commit_transaction(struct btrfs_fs_info *fs_info, return -ENOSPC; spin_lock(&delayed_rsv->lock); - if (delayed_rsv->size > bytes) - bytes = 0; - else - bytes -= delayed_rsv->size; + reclaim_bytes += delayed_rsv->reserved; spin_unlock(&delayed_rsv->lock); + spin_lock(&delayed_refs_rsv->lock); + reclaim_bytes += delayed_refs_rsv->reserved; + spin_unlock(&delayed_refs_rsv->lock); + if (reclaim_bytes >= bytes_needed) + goto commit; + bytes_needed -= reclaim_bytes; + if (__percpu_counter_compare(&space_info->total_bytes_pinned, - bytes, + bytes_needed, BTRFS_TOTAL_BYTES_PINNED_BATCH) < 0) { return -ENOSPC; } From patchwork Mon Dec 3 15:20:35 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 10709809 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 12C5B14E2 for ; Mon, 3 Dec 2018 15:20:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 01DC02A493 for ; Mon, 3 Dec 2018 15:20:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F41242B253; Mon, 3 Dec 2018 15:20:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 854C22B253 for ; Mon, 3 Dec 2018 15:20:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726712AbeLCPU4 (ORCPT ); Mon, 3 Dec 2018 10:20:56 -0500 Received: from mail-yb1-f196.google.com ([209.85.219.196]:39880 "EHLO mail-yb1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726701AbeLCPU4 (ORCPT ); Mon, 3 Dec 2018 10:20:56 -0500 Received: by mail-yb1-f196.google.com with SMTP id w17-v6so5412877ybl.6 for ; Mon, 03 Dec 2018 07:20:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references; bh=39z1AfoM6BOmXC4kNZBHFD4W9bTksWGoXViO/fokQuo=; b=zbkMj1ZrYnCqqUcW9l+C/5jKLzUxIMG/6rZmzf4EkJiPLu+jf7kEcIGin+jZDHuthR 9hJPZhRht6NU7bcbSfDNjOe/rMekxO5gx8UfeypzvFI8WPYe9V8LFDgL1jKZo8e3bmh+ KfLQlRRUgGQD6htkKu1Xf57pJWixQInJM8Hh6+8fT8sbvNRt1TN5JjfCZazCOBouCf6Y BeLbtaC/e1tucNToPrbuFp7GkveVuJ38OqaPQcAqj+qxZ+jp5X4CgwUX/SHJ91ftlleP eVPsUq9Z1ZT0S72R3GjRXcVTw+jGV1AczXNfoasceQLZuzfEs13j23P39MfbkTxbDUsg RCgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=39z1AfoM6BOmXC4kNZBHFD4W9bTksWGoXViO/fokQuo=; b=J0mAN1CewJ7ETxdgXIHwlyLJZFtl6LNH6+kVZt4ZG4KC9tDED1VrlITEckNNofWPpm 4jkEiwRzoJ//cIWW/+mMM8leLy9NhqzSVcY3DbXjnKxIgWrE1NGsXNGtBR4MGnSScueI tdFSyXh8b7fVO9mVYgHihyaMUcAGES2pRlo6BsQpxOiuQGbGBimqVoSwcLe2DrYln9MJ KdtE9vqV5jy3gLv+myRQXCjLbxyaV0fw1mjoHrqZhKSAfRYSztlNse9Mx8/PwTrdqRSq y2LJIrARVuFFMesi+bh7Qpfg+TGEGxiRDPAmOlbKQp14tnFHVPHNuiKQ6p2MG83uls0M E3ag== X-Gm-Message-State: AA+aEWaF2g3+5ag831Ja2C1KwCsTbhgfGp40uqMkXoDLno5KTE4w6JbA hEPGNfh6p6r1LDAq9oCL6favOo5y0lo= X-Google-Smtp-Source: AFSGD/Wgeo4gbPu/kByB8TrMzKlwaYEBBG3pUnS6So7M0Iwgzjhl9cb1Zq+uKueuKV4GruFvDPVAxg== X-Received: by 2002:a25:cbcb:: with SMTP id b194-v6mr15609784ybg.96.1543850454128; Mon, 03 Dec 2018 07:20:54 -0800 (PST) Received: from localhost ([107.15.81.208]) by smtp.gmail.com with ESMTPSA id r62sm4588140ywd.4.2018.12.03.07.20.53 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 03 Dec 2018 07:20:53 -0800 (PST) From: Josef Bacik To: linux-btrfs@vger.kernel.org, kernel-team@fb.com Subject: [PATCH 07/10] btrfs: add new flushing states for the delayed refs rsv Date: Mon, 3 Dec 2018 10:20:35 -0500 Message-Id: <20181203152038.21388-8-josef@toxicpanda.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20181203152038.21388-1-josef@toxicpanda.com> References: <20181203152038.21388-1-josef@toxicpanda.com> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP A nice thing we gain with the delayed refs rsv is the ability to flush the delayed refs on demand to deal with enospc pressure. Add states to flush delayed refs on demand, and this will allow us to remove a lot of ad-hoc work around checking to see if we should commit the transaction to run our delayed refs. Signed-off-by: Josef Bacik --- fs/btrfs/ctree.h | 10 ++++++---- fs/btrfs/extent-tree.c | 14 ++++++++++++++ include/trace/events/btrfs.h | 2 ++ 3 files changed, 22 insertions(+), 4 deletions(-) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h index 52a87d446945..2eba398c722b 100644 --- a/fs/btrfs/ctree.h +++ b/fs/btrfs/ctree.h @@ -2745,10 +2745,12 @@ enum btrfs_reserve_flush_enum { enum btrfs_flush_state { FLUSH_DELAYED_ITEMS_NR = 1, FLUSH_DELAYED_ITEMS = 2, - FLUSH_DELALLOC = 3, - FLUSH_DELALLOC_WAIT = 4, - ALLOC_CHUNK = 5, - COMMIT_TRANS = 6, + FLUSH_DELAYED_REFS_NR = 3, + FLUSH_DELAYED_REFS = 4, + FLUSH_DELALLOC = 5, + FLUSH_DELALLOC_WAIT = 6, + ALLOC_CHUNK = 7, + COMMIT_TRANS = 8, }; int btrfs_alloc_data_chunk_ondemand(struct btrfs_inode *inode, u64 bytes); diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index 63ff9d832867..5a2d0b061f57 100644 --- a/fs/btrfs/extent-tree.c +++ b/fs/btrfs/extent-tree.c @@ -4938,6 +4938,20 @@ static void flush_space(struct btrfs_fs_info *fs_info, shrink_delalloc(fs_info, num_bytes * 2, num_bytes, state == FLUSH_DELALLOC_WAIT); break; + case FLUSH_DELAYED_REFS_NR: + case FLUSH_DELAYED_REFS: + trans = btrfs_join_transaction(root); + if (IS_ERR(trans)) { + ret = PTR_ERR(trans); + break; + } + if (state == FLUSH_DELAYED_REFS_NR) + nr = calc_reclaim_items_nr(fs_info, num_bytes); + else + nr = 0; + btrfs_run_delayed_refs(trans, nr); + btrfs_end_transaction(trans); + break; case ALLOC_CHUNK: trans = btrfs_join_transaction(root); if (IS_ERR(trans)) { diff --git a/include/trace/events/btrfs.h b/include/trace/events/btrfs.h index 8568946f491d..63d1f9d8b8c7 100644 --- a/include/trace/events/btrfs.h +++ b/include/trace/events/btrfs.h @@ -1048,6 +1048,8 @@ TRACE_EVENT(btrfs_trigger_flush, { FLUSH_DELAYED_ITEMS, "FLUSH_DELAYED_ITEMS"}, \ { FLUSH_DELALLOC, "FLUSH_DELALLOC"}, \ { FLUSH_DELALLOC_WAIT, "FLUSH_DELALLOC_WAIT"}, \ + { FLUSH_DELAYED_REFS_NR, "FLUSH_DELAYED_REFS_NR"}, \ + { FLUSH_DELAYED_REFS, "FLUSH_ELAYED_REFS"}, \ { ALLOC_CHUNK, "ALLOC_CHUNK"}, \ { COMMIT_TRANS, "COMMIT_TRANS"}) From patchwork Mon Dec 3 15:20:36 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 10709811 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8723814E2 for ; Mon, 3 Dec 2018 15:20:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 792D42B3FE for ; Mon, 3 Dec 2018 15:20:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6CED42B424; Mon, 3 Dec 2018 15:20:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DD9C22B3E3 for ; Mon, 3 Dec 2018 15:20:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726715AbeLCPU6 (ORCPT ); Mon, 3 Dec 2018 10:20:58 -0500 Received: from mail-yw1-f66.google.com ([209.85.161.66]:40058 "EHLO mail-yw1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726709AbeLCPU5 (ORCPT ); Mon, 3 Dec 2018 10:20:57 -0500 Received: by mail-yw1-f66.google.com with SMTP id r130so5515195ywg.7 for ; Mon, 03 Dec 2018 07:20:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references; bh=Cr1m1zvL9JLqyKDwy0q2EVm1GLsJx38am63DnjAxXpI=; b=W07ErAjsvg3Awji7G8RtZ+wFsElDafrxg1PaK7DsVKo+pYVLOuhoyX0xppwpFsh6D+ AfjIUTXvnYWWMT7Tqmlh5BAnHfog0ejqW3IOWBpefLSiKgUzrikUOG+a8LFwCW5gUEgi vLKVkR2k0DT1yG+9nsp1kaoqo1Xmc+wZSQPUqt9XhE/fBa8iF+cqRnQxw5Y+HcEcDpAt FDnxF458XkwNkEZynsDzvew+U/nOx8Vse9xNxQRMH2TChpmGP6rZhIBIlKiW12FTIzeH 0Fw8Mi0dX1Hk2LUhDlujV0qH379BO7tFv9F3jmu+oW93dn/FoG558Eyw3xdIJAhDmW43 zO5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=Cr1m1zvL9JLqyKDwy0q2EVm1GLsJx38am63DnjAxXpI=; b=G8zIGMwkBvzUIVfHoP94C3cyTX8FfAzWGgaCQmZpmC02/zvxXV7JVavO8hvneVrFi4 2OxKZ5zgKbCTTUdYJpRPkrUWXyUISWno9KuWyMP0ky+ezaoAX/4qICXrDqzGnqg+pS6h gpMk0H2qhpxtwEau5xeP22Yn8Q3SkIqNVfASvgPsd6cg/X1ATqkKIAkD6A23k87ZeWAb W5EVqk/tpiq33DmeAFqAJzW8i0PC0dyV/VKZnvgk2LyC0n4h95yO0iOo4I9hu8731VsA mbqhOFloXByDmZ9B2EvJXQepunL3RDQLjwqnTWOE0nwYTvbvHNZGZD9kxN9yE2KdBM0h zqfQ== X-Gm-Message-State: AA+aEWbXyvmOXFibqJZZS+1+PYOa+DmuAiK4rILlPkz5EhHiPYBy5V7b meoR0pe1cGqKRYLD5NK6eA4r5NJQzJs= X-Google-Smtp-Source: AFSGD/WrAMSuL9GEdO9f/BzLbPfQTS2xcmMDOWBP6N6wiWjjD83H1mo39HSCi8crmMljeEOv1dmA6w== X-Received: by 2002:a81:af68:: with SMTP id x40mr16210615ywj.372.1543850455601; Mon, 03 Dec 2018 07:20:55 -0800 (PST) Received: from localhost ([107.15.81.208]) by smtp.gmail.com with ESMTPSA id n65sm4425339ywf.15.2018.12.03.07.20.54 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 03 Dec 2018 07:20:55 -0800 (PST) From: Josef Bacik To: linux-btrfs@vger.kernel.org, kernel-team@fb.com Subject: [PATCH 08/10] btrfs: rework btrfs_check_space_for_delayed_refs Date: Mon, 3 Dec 2018 10:20:36 -0500 Message-Id: <20181203152038.21388-9-josef@toxicpanda.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20181203152038.21388-1-josef@toxicpanda.com> References: <20181203152038.21388-1-josef@toxicpanda.com> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Now with the delayed_refs_rsv we can now know exactly how much pending delayed refs space we need. This means we can drastically simplify btrfs_check_space_for_delayed_refs by simply checking how much space we have reserved for the global rsv (which acts as a spill over buffer) and the delayed refs rsv. If our total size is beyond that amount then we know it's time to commit the transaction and stop any more delayed refs from being generated. Signed-off-by: Josef Bacik Reviewed-by: Nikolay Borisov --- fs/btrfs/ctree.h | 2 +- fs/btrfs/extent-tree.c | 48 ++++++++++++++++++------------------------------ fs/btrfs/inode.c | 4 ++-- fs/btrfs/transaction.c | 2 +- 4 files changed, 22 insertions(+), 34 deletions(-) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h index 2eba398c722b..30da075c042e 100644 --- a/fs/btrfs/ctree.h +++ b/fs/btrfs/ctree.h @@ -2631,7 +2631,7 @@ static inline u64 btrfs_calc_trunc_metadata_size(struct btrfs_fs_info *fs_info, } int btrfs_should_throttle_delayed_refs(struct btrfs_trans_handle *trans); -int btrfs_check_space_for_delayed_refs(struct btrfs_trans_handle *trans); +bool btrfs_check_space_for_delayed_refs(struct btrfs_fs_info *fs_info); void btrfs_dec_block_group_reservations(struct btrfs_fs_info *fs_info, const u64 start); void btrfs_wait_block_group_reservations(struct btrfs_block_group_cache *bg); diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index 5a2d0b061f57..07ef1b8087f7 100644 --- a/fs/btrfs/extent-tree.c +++ b/fs/btrfs/extent-tree.c @@ -2839,40 +2839,28 @@ u64 btrfs_csum_bytes_to_leaves(struct btrfs_fs_info *fs_info, u64 csum_bytes) return num_csums; } -int btrfs_check_space_for_delayed_refs(struct btrfs_trans_handle *trans) +bool btrfs_check_space_for_delayed_refs(struct btrfs_fs_info *fs_info) { - struct btrfs_fs_info *fs_info = trans->fs_info; - struct btrfs_block_rsv *global_rsv; - u64 num_heads = trans->transaction->delayed_refs.num_heads_ready; - u64 csum_bytes = trans->transaction->delayed_refs.pending_csums; - unsigned int num_dirty_bgs = trans->transaction->num_dirty_bgs; - u64 num_bytes, num_dirty_bgs_bytes; - int ret = 0; + struct btrfs_block_rsv *delayed_refs_rsv = &fs_info->delayed_refs_rsv; + struct btrfs_block_rsv *global_rsv = &fs_info->global_block_rsv; + bool ret = false; + u64 reserved; - num_bytes = btrfs_calc_trans_metadata_size(fs_info, 1); - num_heads = heads_to_leaves(fs_info, num_heads); - if (num_heads > 1) - num_bytes += (num_heads - 1) * fs_info->nodesize; - num_bytes <<= 1; - num_bytes += btrfs_csum_bytes_to_leaves(fs_info, csum_bytes) * - fs_info->nodesize; - num_dirty_bgs_bytes = btrfs_calc_trans_metadata_size(fs_info, - num_dirty_bgs); - global_rsv = &fs_info->global_block_rsv; + spin_lock(&global_rsv->lock); + reserved = global_rsv->reserved; + spin_unlock(&global_rsv->lock); /* - * If we can't allocate any more chunks lets make sure we have _lots_ of - * wiggle room since running delayed refs can create more delayed refs. + * Since the global reserve is just kind of magic we don't really want + * to rely on it to save our bacon, so if our size is more than the + * delayed_refs_rsv and the global rsv then it's time to think about + * bailing. */ - if (global_rsv->space_info->full) { - num_dirty_bgs_bytes <<= 1; - num_bytes <<= 1; - } - - spin_lock(&global_rsv->lock); - if (global_rsv->reserved <= num_bytes + num_dirty_bgs_bytes) - ret = 1; - spin_unlock(&global_rsv->lock); + spin_lock(&delayed_refs_rsv->lock); + reserved += delayed_refs_rsv->reserved; + if (delayed_refs_rsv->size >= reserved) + ret = true; + spin_unlock(&delayed_refs_rsv->lock); return ret; } @@ -2891,7 +2879,7 @@ int btrfs_should_throttle_delayed_refs(struct btrfs_trans_handle *trans) if (val >= NSEC_PER_SEC / 2) return 2; - return btrfs_check_space_for_delayed_refs(trans); + return btrfs_check_space_for_delayed_refs(trans->fs_info); } struct async_delayed_refs { diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index a097f5fde31d..8532a2eb56d1 100644 --- a/fs/btrfs/inode.c +++ b/fs/btrfs/inode.c @@ -5326,8 +5326,8 @@ static struct btrfs_trans_handle *evict_refill_and_join(struct btrfs_root *root, * Try to steal from the global reserve if there is space for * it. */ - if (!btrfs_check_space_for_delayed_refs(trans) && - !btrfs_block_rsv_migrate(global_rsv, rsv, rsv->size, false)) + if (!btrfs_check_space_for_delayed_refs(fs_info) && + !btrfs_block_rsv_migrate(global_rsv, rsv, rsv->size, 0)) return trans; /* If not, commit and try again. */ diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c index a21c4defad92..2d8401bf8df9 100644 --- a/fs/btrfs/transaction.c +++ b/fs/btrfs/transaction.c @@ -789,7 +789,7 @@ static int should_end_transaction(struct btrfs_trans_handle *trans) { struct btrfs_fs_info *fs_info = trans->fs_info; - if (btrfs_check_space_for_delayed_refs(trans)) + if (btrfs_check_space_for_delayed_refs(fs_info)) return 1; return !!btrfs_block_rsv_check(&fs_info->global_block_rsv, 5); From patchwork Mon Dec 3 15:20:37 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 10709813 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 95AE014E2 for ; Mon, 3 Dec 2018 15:21:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8706C2B271 for ; Mon, 3 Dec 2018 15:21:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 84EE52B25C; Mon, 3 Dec 2018 15:21:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1D94E2B271 for ; Mon, 3 Dec 2018 15:21:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726721AbeLCPU7 (ORCPT ); Mon, 3 Dec 2018 10:20:59 -0500 Received: from mail-yw1-f65.google.com ([209.85.161.65]:41816 "EHLO mail-yw1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726709AbeLCPU7 (ORCPT ); Mon, 3 Dec 2018 10:20:59 -0500 Received: by mail-yw1-f65.google.com with SMTP id f65so5516073ywc.8 for ; Mon, 03 Dec 2018 07:20:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references; bh=hHtcDYnXOhWCyKFQ33WPBhLZWr2Xg4OGIItW/Kobk30=; b=BsrqaSo7pVGGHf+2VXC00+hwJMuZwugOxF0OriBvRjn2GsJBTmlnshotCn1k3oUQyd aMfFCbHSco/93E2tRkH781QsX5dFKiWp137SzLGvO0zauMGUPbSWDBQQ2we7WOqdqzpB HjcCNGuUR8QmEfHhRivchLWySKchbFY2uIQXv28Z4w0DeuHcVVAnbkdgEgAYYE8PM/rV C9FKT7uHwZIUWQEz2HQNpRiXVTL1Ak4US6Q4vd4imFquGzSDNvxu+KXDIsgveLsaZh8r HSeToMFCJeZ2il9b0DQGWZOtW3AVokuvx6OzJSlzPyPEnTe1kTzNweB8qvdGd55daXj0 aAYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=hHtcDYnXOhWCyKFQ33WPBhLZWr2Xg4OGIItW/Kobk30=; b=i6BwEcElZJl3wRtFs0+sx2LDHYWQ2oGdCsZbHXmJutlMVyY/o0mGvNXzfw4mFRMahQ M8XyixIjWMkJUS5MZfiPFrmlrcJkYmA1mdc5+k8JGd7HmaoByQKWQtkEf4G5Hva3qrD2 y8+pn8khIDP24N7XpPcZ4ZbZ0hAd3Uz6/TXMzB+L6c1P0T2teYMOuLomgXnb1CaP8HzM /80lScePG0g7N7/0gdJc6uiSXQRQkhvFVqTJqc3+3kliFLE1AedpWhoCFLLvmMRPc3M4 I+a4XVE9xVR8e5gD0c8u/9tpGoZ8x6L0MvSPtp3cxVSBi1Khhpher7yZ1swYSSpq2dUt nVQA== X-Gm-Message-State: AA+aEWZuTQRB7xfSjKpXJnLvnMsGorBHCPrp4t2mBRjk1ItziIZ6NkmI Ia3DTovFfaZCF072/Kon61LaikDdvjg= X-Google-Smtp-Source: AFSGD/W1hlx8TUeYPw0fheObuJRja+w4tHfwYtrQQ+m8NOtfwstgv0bp0ouIrs7CTphrRZRiqcMB8Q== X-Received: by 2002:a81:b65d:: with SMTP id h29mr15609321ywk.429.1543850457285; Mon, 03 Dec 2018 07:20:57 -0800 (PST) Received: from localhost ([107.15.81.208]) by smtp.gmail.com with ESMTPSA id p201sm5792670ywe.45.2018.12.03.07.20.56 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 03 Dec 2018 07:20:56 -0800 (PST) From: Josef Bacik To: linux-btrfs@vger.kernel.org, kernel-team@fb.com Subject: [PATCH 09/10] btrfs: don't run delayed refs in the end transaction logic Date: Mon, 3 Dec 2018 10:20:37 -0500 Message-Id: <20181203152038.21388-10-josef@toxicpanda.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20181203152038.21388-1-josef@toxicpanda.com> References: <20181203152038.21388-1-josef@toxicpanda.com> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Over the years we have built up a lot of infrastructure to keep delayed refs in check, mostly by running them at btrfs_end_transaction() time. We have a lot of different maths we do to figure out how much, if we should do it inline or async, etc. This existed because we had no feedback mechanism to force the flushing of delayed refs when they became a problem. However with the enospc flushing infrastructure in place for flushing delayed refs when they put too much pressure on the enospc system we have this problem solved. Rip out all of this code as it is no longer needed. Signed-off-by: Josef Bacik --- fs/btrfs/transaction.c | 38 -------------------------------------- 1 file changed, 38 deletions(-) diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c index 2d8401bf8df9..01f39401619a 100644 --- a/fs/btrfs/transaction.c +++ b/fs/btrfs/transaction.c @@ -798,22 +798,12 @@ static int should_end_transaction(struct btrfs_trans_handle *trans) int btrfs_should_end_transaction(struct btrfs_trans_handle *trans) { struct btrfs_transaction *cur_trans = trans->transaction; - int updates; - int err; smp_mb(); if (cur_trans->state >= TRANS_STATE_BLOCKED || cur_trans->delayed_refs.flushing) return 1; - updates = trans->delayed_ref_updates; - trans->delayed_ref_updates = 0; - if (updates) { - err = btrfs_run_delayed_refs(trans, updates * 2); - if (err) /* Error code will also eval true */ - return err; - } - return should_end_transaction(trans); } @@ -843,11 +833,8 @@ static int __btrfs_end_transaction(struct btrfs_trans_handle *trans, { struct btrfs_fs_info *info = trans->fs_info; struct btrfs_transaction *cur_trans = trans->transaction; - u64 transid = trans->transid; - unsigned long cur = trans->delayed_ref_updates; int lock = (trans->type != TRANS_JOIN_NOLOCK); int err = 0; - int must_run_delayed_refs = 0; if (refcount_read(&trans->use_count) > 1) { refcount_dec(&trans->use_count); @@ -858,27 +845,6 @@ static int __btrfs_end_transaction(struct btrfs_trans_handle *trans, btrfs_trans_release_metadata(trans); trans->block_rsv = NULL; - if (!list_empty(&trans->new_bgs)) - btrfs_create_pending_block_groups(trans); - - trans->delayed_ref_updates = 0; - if (!trans->sync) { - must_run_delayed_refs = - btrfs_should_throttle_delayed_refs(trans); - cur = max_t(unsigned long, cur, 32); - - /* - * don't make the caller wait if they are from a NOLOCK - * or ATTACH transaction, it will deadlock with commit - */ - if (must_run_delayed_refs == 1 && - (trans->type & (__TRANS_JOIN_NOLOCK | __TRANS_ATTACH))) - must_run_delayed_refs = 2; - } - - btrfs_trans_release_metadata(trans); - trans->block_rsv = NULL; - if (!list_empty(&trans->new_bgs)) btrfs_create_pending_block_groups(trans); @@ -923,10 +889,6 @@ static int __btrfs_end_transaction(struct btrfs_trans_handle *trans, } kmem_cache_free(btrfs_trans_handle_cachep, trans); - if (must_run_delayed_refs) { - btrfs_async_run_delayed_refs(info, cur, transid, - must_run_delayed_refs == 1); - } return err; } From patchwork Mon Dec 3 15:20:38 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 10709815 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EB0CA17D5 for ; Mon, 3 Dec 2018 15:21:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DB9B82B402 for ; Mon, 3 Dec 2018 15:21:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CFC532B3FE; Mon, 3 Dec 2018 15:21:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C02C72B297 for ; Mon, 3 Dec 2018 15:21:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726724AbeLCPVB (ORCPT ); Mon, 3 Dec 2018 10:21:01 -0500 Received: from mail-yb1-f196.google.com ([209.85.219.196]:43021 "EHLO mail-yb1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726709AbeLCPVA (ORCPT ); Mon, 3 Dec 2018 10:21:00 -0500 Received: by mail-yb1-f196.google.com with SMTP id x144so1256117ybg.10 for ; Mon, 03 Dec 2018 07:20:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references; bh=7pkuIGhBeFNoexUWutrfJ+o+6+fDIQj9nz0xcSK0ItI=; b=T8eDwPeKVEDYoa0A/QHJWJ5aMqTJlGibU9THDACpUESRcPwhUcTQDIKYQwmhgKrOmE CflN2TwtRL5YEFc4PMmINkZUywVc8Rw/rI+qKwd6566g+FqmEdiKBYvH06ErdbMYR9/o z4ocUb4R5pogCIuVMz7yZEsbv70GKD6DIb32D0GDFWA6xTPm7I9roEKpNUZjuzPhTt9q i6s3cuAGOseOPCH/OmixnyBgrr7C/w+5GjA+rOgS3Bf3UBRObgH9nINqWLz5XnfZaOoc MN1Vy5/iwJGOVhLEJt+i/WLLn8v60eAHBqNA1LFN9B0pPJ3oxKlh+f/WXuFcj4kuppQj 9AQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=7pkuIGhBeFNoexUWutrfJ+o+6+fDIQj9nz0xcSK0ItI=; b=jxViA4jXePCenWoPrayMkVBfnPb2lIjcI00L7SIU8HdfEtBCsTzAF4pKZtRPRz53qy S3qQGghu4zslJzTaM4ldd2fZGFjdy6XBiJBqaK0FG437pXt2AuzmDqMSfFCnJTlY4YNS T0PuGwlvWRCAy/xhSk4qvWMcXpkrRz8J9+Eh8tK7kq9bj2ZL7cyhzCigrOoBNMScPRaX 1+Xu/757zSVzynU43bDP1HOVBJQ+P/7YB5MWqKvBIpDbz0ly3BqQ6BuW9zOi6cfvfgPO yHxHp9XMksQLYs53H9li15+/Ud+sXqwIHaLxi5FHCK9i3ksqaksw2gNinZZYpwfhWIW+ yOrw== X-Gm-Message-State: AA+aEWbWbEsOrVDFktNDoyvJsv62A6egtkMiHZAcDGc2WAkCn3NeXXwQ 2AHRvoakK9esaG9pqzEHg9vwtXS+dmQ= X-Google-Smtp-Source: AFSGD/W3ab+bB+dG401i018CZgl/+710SO+ZOPIEJ1ifyD/U7RWgr5pSmvDueDnsdT7GKmwjlEfAnQ== X-Received: by 2002:a25:3c43:: with SMTP id j64-v6mr15761303yba.517.1543850458838; Mon, 03 Dec 2018 07:20:58 -0800 (PST) Received: from localhost ([107.15.81.208]) by smtp.gmail.com with ESMTPSA id 136sm5176831ywm.28.2018.12.03.07.20.57 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 03 Dec 2018 07:20:58 -0800 (PST) From: Josef Bacik To: linux-btrfs@vger.kernel.org, kernel-team@fb.com Subject: [PATCH 10/10] btrfs: fix truncate throttling Date: Mon, 3 Dec 2018 10:20:38 -0500 Message-Id: <20181203152038.21388-11-josef@toxicpanda.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20181203152038.21388-1-josef@toxicpanda.com> References: <20181203152038.21388-1-josef@toxicpanda.com> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP We have a bunch of magic to make sure we're throttling delayed refs when truncating a file. Now that we have a delayed refs rsv and a mechanism for refilling that reserve simply use that instead of all of this magic. Reviewed-by: Nikolay Borisov Signed-off-by: Josef Bacik --- fs/btrfs/inode.c | 79 ++++++++++++-------------------------------------------- 1 file changed, 17 insertions(+), 62 deletions(-) diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index 8532a2eb56d1..623a71d871d4 100644 --- a/fs/btrfs/inode.c +++ b/fs/btrfs/inode.c @@ -4437,31 +4437,6 @@ static int btrfs_rmdir(struct inode *dir, struct dentry *dentry) return err; } -static int truncate_space_check(struct btrfs_trans_handle *trans, - struct btrfs_root *root, - u64 bytes_deleted) -{ - struct btrfs_fs_info *fs_info = root->fs_info; - int ret; - - /* - * This is only used to apply pressure to the enospc system, we don't - * intend to use this reservation at all. - */ - bytes_deleted = btrfs_csum_bytes_to_leaves(fs_info, bytes_deleted); - bytes_deleted *= fs_info->nodesize; - ret = btrfs_block_rsv_add(root, &fs_info->trans_block_rsv, - bytes_deleted, BTRFS_RESERVE_NO_FLUSH); - if (!ret) { - trace_btrfs_space_reservation(fs_info, "transaction", - trans->transid, - bytes_deleted, 1); - trans->bytes_reserved += bytes_deleted; - } - return ret; - -} - /* * Return this if we need to call truncate_block for the last bit of the * truncate. @@ -4506,7 +4481,6 @@ int btrfs_truncate_inode_items(struct btrfs_trans_handle *trans, u64 bytes_deleted = 0; bool be_nice = false; bool should_throttle = false; - bool should_end = false; BUG_ON(new_size > 0 && min_type != BTRFS_EXTENT_DATA_KEY); @@ -4719,15 +4693,7 @@ int btrfs_truncate_inode_items(struct btrfs_trans_handle *trans, btrfs_abort_transaction(trans, ret); break; } - if (btrfs_should_throttle_delayed_refs(trans)) - btrfs_async_run_delayed_refs(fs_info, - trans->delayed_ref_updates * 2, - trans->transid, 0); if (be_nice) { - if (truncate_space_check(trans, root, - extent_num_bytes)) { - should_end = true; - } if (btrfs_should_throttle_delayed_refs(trans)) should_throttle = true; } @@ -4738,7 +4704,7 @@ int btrfs_truncate_inode_items(struct btrfs_trans_handle *trans, if (path->slots[0] == 0 || path->slots[0] != pending_del_slot || - should_throttle || should_end) { + should_throttle) { if (pending_del_nr) { ret = btrfs_del_items(trans, root, path, pending_del_slot, @@ -4750,23 +4716,24 @@ int btrfs_truncate_inode_items(struct btrfs_trans_handle *trans, pending_del_nr = 0; } btrfs_release_path(path); - if (should_throttle) { - unsigned long updates = trans->delayed_ref_updates; - if (updates) { - trans->delayed_ref_updates = 0; - ret = btrfs_run_delayed_refs(trans, - updates * 2); - if (ret) - break; - } - } + /* - * if we failed to refill our space rsv, bail out - * and let the transaction restart + * We can generate a lot of delayed refs, so we need to + * throttle every once and a while and make sure we're + * adding enough space to keep up with the work we are + * generating. Since we hold a transaction here we + * can't flush, and we don't want to FLUSH_LIMIT because + * we could have generated too many delayed refs to + * actually allocate, so just bail if we're short and + * let the normal reservation dance happen higher up. */ - if (should_end) { - ret = -EAGAIN; - break; + if (should_throttle) { + ret = btrfs_delayed_refs_rsv_refill(fs_info, + BTRFS_RESERVE_NO_FLUSH); + if (ret) { + ret = -EAGAIN; + break; + } } goto search_again; } else { @@ -4792,18 +4759,6 @@ int btrfs_truncate_inode_items(struct btrfs_trans_handle *trans, } btrfs_free_path(path); - - if (be_nice && bytes_deleted > SZ_32M && (ret >= 0 || ret == -EAGAIN)) { - unsigned long updates = trans->delayed_ref_updates; - int err; - - if (updates) { - trans->delayed_ref_updates = 0; - err = btrfs_run_delayed_refs(trans, updates * 2); - if (err) - ret = err; - } - } return ret; }