From patchwork Fri Sep 22 01:43:04 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brad Hubbard X-Patchwork-Id: 9965021 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id C944D601D5 for ; Fri, 22 Sep 2017 01:43:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B3F5B29645 for ; Fri, 22 Sep 2017 01:43:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A8E5629652; Fri, 22 Sep 2017 01:43:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00, MSGID_FROM_MTA_HEADER,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6CB0B29645 for ; Fri, 22 Sep 2017 01:43:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751800AbdIVBnM (ORCPT ); Thu, 21 Sep 2017 21:43:12 -0400 Received: from mail-pg0-f50.google.com ([74.125.83.50]:44020 "EHLO mail-pg0-f50.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751790AbdIVBnL (ORCPT ); Thu, 21 Sep 2017 21:43:11 -0400 Received: by mail-pg0-f50.google.com with SMTP id u18so4508140pgo.0 for ; Thu, 21 Sep 2017 18:43:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=zpwNM2d9Big7JRvC8jQarzav8+ejWyQfeneh8XJxmtk=; b=nQiEQck2CDa7pHR+oty0rdIifSoPBHd1MoCtmCr0KJ36Sgo8SKMMGpHVpc2BQAVtSx 0Vf4WWAKg/kWXsUL/39KYvlFD33fM0BLQCIsNIUV42kIumTdjyhfVElhCXSbxV4zT6fA uO7YzfrnzA67Kdeo+eRCvjAYmT2xBM8sCW4Akhu7Zb7Ujt14HuVxJxgnDyFFxmz9ZzTA AOZnM8QR7hAMs1MSIzccZobe584Cdqlyh97WOlcjQz5u8e+9C6WfNbjirrqv0M7OrfFB nuobk5j+ig8weoYHQ6NaD5P82SuDkTia9LXFZKg6xEhmyIMnqfDOBcjOdAtpZ9qv/oPw E71Q== X-Gm-Message-State: AHPjjUh4SzbInw2CdezkZQJjV6qTblIErFIWkNrk8mBsxheiwIMgD7f/ J3TU1RumrhYn5UPRk0jRdOr2KQ== X-Google-Smtp-Source: AOwi7QDfXw5XBN2qK1AvijSvLKsDR+jCi0bTpTG7jMvKwNoSvLhX48Y4vcJ4pYraILbtk4cDyvCUgw== X-Received: by 10.98.150.27 with SMTP id c27mr7673737pfe.123.1506044590657; Thu, 21 Sep 2017 18:43:10 -0700 (PDT) Received: from rskikr.localdomain (124-171-93-177.dyn.iinet.net.au. [124.171.93.177]) by smtp.gmail.com with ESMTPSA id z26sm5831987pfa.49.2017.09.21.18.43.08 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 21 Sep 2017 18:43:09 -0700 (PDT) Date: Fri, 22 Sep 2017 11:43:04 +1000 From: Brad Hubbard To: kefu chai Cc: "ceph-devel@vger.kernel.org" Subject: Re: fix for "crash in rocksdb LRUCache destructor with tcmalloc" Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Received: by 10.79.158.71 with HTTP; Wed, 20 Sep 2017 17:40:28 -0700 (PDT) User-Agent: Mutt/1.8.3 (2017-05-23) Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Wed, Sep 20, 2017 at 12:58 PM, kefu chai wrote: Thanks for posting this Kefu as well as taking the lead on diagnosing this. Just want to clarify some details below. > hi gperf-tools v2.5.93 (or fc25 users), Think you mean fc26 users? > > we sync'ed rocksdb to the recent upstream version in master last > week-end. if you are using gperf-tools 2.5.93 (which is shipped with > fc25), then you are likely to hit 2.5.93 is shipped with fc26. > http://tracker.ceph.com/issues/21422. the root cause is that tcmalloc > is offered by gperf-tools. but gperf-tools v2.5.93 didn't implement > aligned_alloc(), and the recent version of rocksdb is using it. so > what gets called is the glibc's aligned_alloc(), and when rocksdb > frees that memory chunk allocated by aligned_alloc(), the free() from > tcmalloc is used. so it panic'ed. for more details, please refer to > the tracker ticket[0] and the pull request[1] to address it. > > if you compile the latest master of ceph using gperf-tools 2.5.93, you will get > > Incompatible tcmalloc v2.5.93 and rocksdb v5.8.0, please install > gperf-tools 2.5 or > 2.6.2". > > since the latest release of gperf-tools is 2.6.1, and it has not yet > cut a release after including the for the aligned_alloc(), currently > the suggested way to fix it is to upgrade your gperf-tools to the > latest master by installing it[3] manually. and point cmake to it: > > $ ./configure --prefix=$HOME/local # under gperftools > $ make install > $ GPERF_ROOT=$HOME/local cmake .. # under ceph/build If you do not encounter dependency hell you can also remove gperftools-devel-2.5.93-1.fc26.x86_64 and gperftools-libs-2.5.93-1.fc26.x86_64 and install gperftools-devel-2.5-2.fc25.x86_64 and gperftools-libs-2.5-2.fc25.x86_64 from http://download.bne.redhat.com/pub/fedora/linux/releases/25/Everything/x86_64/os/Packages/g/ as I have had success up to this point with those packages (YMMV). Of course as Jeff pointed out you can also specify the libc allocator to cmake. Alternatively, I have created an unofficial package that contains the required patch here. https://copr-be.cloud.fedoraproject.org/results/badone/misc/fedora-26-x86_64/00606401-gperftools/ So far this seems to work fine but it may cause problems so caveat emptor :) This patch should allow you to build with my package. I've also created a bug against fc26 here. https://bugzilla.redhat.com/show_bug.cgi?id=1494309 > > > > --- > [0] http://tracker.ceph.com/issues/21422 > [1] https://github.com/ceph/ceph/pull/17788 > [3] https://github.com/gperftools/gperftools/tree/master > > -- > Regards > Kefu Chai > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html --- Cheers, Brad -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/cmake/modules/BuildRocksDB.cmake b/cmake/modules/BuildRocksDB.cmake index 8c44098d6f..efbd97fab2 100644 --- a/cmake/modules/BuildRocksDB.cmake +++ b/cmake/modules/BuildRocksDB.cmake @@ -59,7 +59,8 @@ macro(build_rocksdb) # see http://tracker.ceph.com/issues/21422 if(ROCKSDB_VERSION_STRING VERSION_GREATER 5.7 AND TCMALLOC_VERSION_STRING VERSION_GREATER 2.5 AND - TCMALLOC_VERSION_STRING VERSION_LESS 2.6.2) + TCMALLOC_VERSION_STRING VERSION_LESS 2.6.2 AND NOT + TCMALLOC_VERSION_STRING VERSION_EQUAL 2.5.93-x) message(SEND_ERROR "Incompatible tcmalloc v${TCMALLOC_VERSION_STRING} and rocksdb v${ROCKSDB_VERSION_STRING}, " "please install gperf-tools 2.5 or > 2.6.2")