From patchwork Thu Aug 28 07:06:19 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Oliva X-Patchwork-Id: 4792881 Return-Path: X-Original-To: patchwork-ceph-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 3C2CFC0338 for ; Thu, 28 Aug 2014 07:08:46 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 26AFF2015A for ; Thu, 28 Aug 2014 07:08:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 4FC1B200E3 for ; Thu, 28 Aug 2014 07:08:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S935680AbaH1HIm (ORCPT ); Thu, 28 Aug 2014 03:08:42 -0400 Received: from linux-libre.fsfla.org ([208.118.235.54]:59844 "EHLO linux-libre.fsfla.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932622AbaH1HIl (ORCPT ); Thu, 28 Aug 2014 03:08:41 -0400 Received: from freie.home (home.lxoliva.fsfla.org [172.31.160.22]) by linux-libre.fsfla.org (8.14.4/8.14.4/Debian-2ubuntu2.1) with ESMTP id s7S78Ynn025124; Thu, 28 Aug 2014 07:08:35 GMT Received: from free.home (free.home [172.31.160.1]) by freie.home (8.14.8/8.14.7) with ESMTP id s7S76K46018328; Thu, 28 Aug 2014 04:06:24 -0300 From: Alexandre Oliva To: Sage Weil Cc: =?utf-8?Q?Micha=C5=82_Szyma=C5=84ski?= , ceph-devel Subject: Re: OSD's load_pgs takes a lot of time Organization: Free thinker, not speaking for the GNU Project References: Date: Thu, 28 Aug 2014 04:06:19 -0300 In-Reply-To: (Sage Weil's message of "Tue, 26 Aug 2014 06:03:43 -0700 (PDT)") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.2 (gnu/linux) MIME-Version: 1.0 Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, T_TVD_MIME_EPI, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Aug 26, 2014, Sage Weil wrote: > On Tue, 26 Aug 2014, Micha? Szyma?ski wrote: >> 2014-08-21 15:32:19.648610 7fba11569780 10 osd.1 139 load_pgs >> 3.13b_TEMP clearing temp >> 2014-08-21 15:32:34.674773 7fba11569780 10 osd.1 139 load_pgs >> 3.36b_TEMP clearing temp >> Have anybody noticed this behavior? Isn't it possible to clear temp faster? > load_pgs is doing a lot more than just clearing temp But at least in some scenarios using btrfs, clearing temps turns out to be the dominant factor, and I posted a patch a while ago that solves that specific case, but I don't think it was ever merged. Micha?, does this help? When a btrfs OSD suicides because it couldn't keep up with others (recovery included), it is often the case that the filesystem is in a state in which creating and removing snapshots takes a long time. It is likely that there are going to be plenty of recovering PGs (with corresponding TEMP directories), and we will create and remove a btrfs snapshot for every such directory we remove at startup. This is very wasteful: there's no reason to not pile up as many removals as possible before syncing and flushing the removal so as to force a btrfs snapshot. This patch introduces an optional argument to the function used to remove PGs to control whether or not to perform the final flush. The flush still happens if the call does not pass that argument, but with the patch, the startup removals skip the per-PG flush, and run a single flush at the end. Signed-off-by: Alexandre Oliva --- src/osd/OSD.cc | 12 +++++++++--- src/osd/OSD.h | 3 ++- 2 files changed, 11 insertions(+), 4 deletions(-) diff --git a/src/osd/OSD.cc b/src/osd/OSD.cc index 4240ba8..504cb71 100644 --- a/src/osd/OSD.cc +++ b/src/osd/OSD.cc @@ -1774,7 +1774,8 @@ int OSD::read_superblock() -void OSD::recursive_remove_collection(ObjectStore *store, coll_t tmp) +void OSD::recursive_remove_collection(ObjectStore *store, coll_t tmp, + bool flush) { OSDriver driver( store, @@ -1810,7 +1811,8 @@ void OSD::recursive_remove_collection(ObjectStore *store, coll_t tmp) t.remove_collection(tmp); int r = store->apply_transaction(t); assert(r == 0); - store->sync_and_flush(); + if (flush) + store->sync_and_flush(); } @@ -2046,6 +2048,7 @@ void OSD::load_pgs() set head_pgs; map > pgs; + bool flush = false; for (vector::iterator it = ls.begin(); it != ls.end(); ++it) { @@ -2056,7 +2059,8 @@ void OSD::load_pgs() if (it->is_temp(pgid) || it->is_removal(&seq, &pgid)) { dout(10) << "load_pgs " << *it << " clearing temp" << dendl; - recursive_remove_collection(store, *it); + recursive_remove_collection(store, *it, false); + flush = true; continue; } @@ -2074,6 +2078,8 @@ void OSD::load_pgs() dout(10) << "load_pgs ignoring unrecognized " << *it << dendl; } + if (flush) + store->sync_and_flush(); bool has_upgraded = false; for (map >::iterator i = pgs.begin(); diff --git a/src/osd/OSD.h b/src/osd/OSD.h index ce8b74c..f599e43 100644 --- a/src/osd/OSD.h +++ b/src/osd/OSD.h @@ -870,7 +870,8 @@ public: hobject_t oid(sobject_t("infos", CEPH_NOSNAP)); return oid; } - static void recursive_remove_collection(ObjectStore *store, coll_t tmp); + static void recursive_remove_collection(ObjectStore *store, coll_t tmp, + bool flush = true); /** * get_osd_initial_compat_set()