fix for "crash in rocksdb LRUCache destructor with tcmalloc"
diff mbox

Message ID CAF-wwdFwU5B327bvQHYXmH8v8+sa1ifVZjg00Aq36OxE15Z6wg@mail.gmail.com
State New
Headers show

Commit Message

Brad Hubbard Sept. 22, 2017, 1:43 a.m. UTC
On Wed, Sep 20, 2017 at 12:58 PM, kefu chai <tchaikov@gmail.com> wrote:

Thanks for posting this Kefu as well as taking the lead on diagnosing this.

Just want to clarify some details below.

> hi gperf-tools v2.5.93 (or fc25 users),

Think you mean fc26 users?

>
> we sync'ed rocksdb to the recent upstream version in master last
> week-end. if you are using gperf-tools 2.5.93 (which is shipped with
> fc25), then you are likely to hit

2.5.93 is shipped with fc26.

> http://tracker.ceph.com/issues/21422. the root cause is that tcmalloc
> is offered by gperf-tools. but gperf-tools v2.5.93 didn't implement
> aligned_alloc(), and the recent version of rocksdb is using it. so
> what gets called is the glibc's aligned_alloc(), and when rocksdb
> frees that memory chunk allocated by aligned_alloc(), the free() from
> tcmalloc is used. so it panic'ed. for more details, please refer to
> the tracker ticket[0] and the pull request[1] to address it.
>
> if you compile the latest master of ceph using gperf-tools 2.5.93, you will get
>
> Incompatible tcmalloc v2.5.93 and rocksdb v5.8.0, please install
> gperf-tools 2.5 or > 2.6.2".
>
> since the latest release of gperf-tools is 2.6.1, and it has not yet
> cut a release after including the for the aligned_alloc(), currently
> the suggested way to fix it is to upgrade your gperf-tools to the
> latest master by installing it[3] manually. and point cmake to it:
>
> $ ./configure --prefix=$HOME/local # under gperftools
> $ make install
> $ GPERF_ROOT=$HOME/local cmake .. # under ceph/build

If you do not encounter dependency hell you can also remove
gperftools-devel-2.5.93-1.fc26.x86_64 and gperftools-libs-2.5.93-1.fc26.x86_64
and install gperftools-devel-2.5-2.fc25.x86_64 and
gperftools-libs-2.5-2.fc25.x86_64 from
http://download.bne.redhat.com/pub/fedora/linux/releases/25/Everything/x86_64/os/Packages/g/
as I have had success up to this point with those packages (YMMV). Of course as
Jeff pointed out you can also specify the libc allocator to cmake.

Alternatively, I have created an unofficial package that contains the required
patch here.

https://copr-be.cloud.fedoraproject.org/results/badone/misc/fedora-26-x86_64/00606401-gperftools/

So far this seems to work fine but it may cause problems so caveat emptor :)

This patch should allow you to build with my package.


I've also created a bug against fc26 here.

https://bugzilla.redhat.com/show_bug.cgi?id=1494309

>
>
>
> ---
> [0] http://tracker.ceph.com/issues/21422
> [1] https://github.com/ceph/ceph/pull/17788
> [3] https://github.com/gperftools/gperftools/tree/master
>
> --
> Regards
> Kefu Chai
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



--
Cheers,
Brad
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Comments

kefu chai Sept. 22, 2017, 6:42 a.m. UTC | #1
On Fri, Sep 22, 2017 at 9:43 AM, Brad Hubbard <bhubbard@redhat.com> wrote:
>
>
> On Wed, Sep 20, 2017 at 12:58 PM, kefu chai <tchaikov@gmail.com> wrote:
>
> Thanks for posting this Kefu as well as taking the lead on diagnosing this.
>
> Just want to clarify some details below.
>
>> hi gperf-tools v2.5.93 (or fc25 users),
>
> Think you mean fc26 users?

yes, please s/fc25/fc26/.

>
>>
>> we sync'ed rocksdb to the recent upstream version in master last
>> week-end. if you are using gperf-tools 2.5.93 (which is shipped with
>> fc25), then you are likely to hit
>
> 2.5.93 is shipped with fc26.
>
>> http://tracker.ceph.com/issues/21422. the root cause is that tcmalloc
>> is offered by gperf-tools. but gperf-tools v2.5.93 didn't implement
>> aligned_alloc(), and the recent version of rocksdb is using it. so
>> what gets called is the glibc's aligned_alloc(), and when rocksdb
>> frees that memory chunk allocated by aligned_alloc(), the free() from
>> tcmalloc is used. so it panic'ed. for more details, please refer to
>> the tracker ticket[0] and the pull request[1] to address it.
>>
>> if you compile the latest master of ceph using gperf-tools 2.5.93, you will get
>>
>> Incompatible tcmalloc v2.5.93 and rocksdb v5.8.0, please install
>> gperf-tools 2.5 or > 2.6.2".
>>
>> since the latest release of gperf-tools is 2.6.1, and it has not yet
>> cut a release after including the for the aligned_alloc(), currently
>> the suggested way to fix it is to upgrade your gperf-tools to the
>> latest master by installing it[3] manually. and point cmake to it:
>>
>> $ ./configure --prefix=$HOME/local # under gperftools
>> $ make install
>> $ GPERF_ROOT=$HOME/local cmake .. # under ceph/build
>
> If you do not encounter dependency hell you can also remove
> gperftools-devel-2.5.93-1.fc26.x86_64 and gperftools-libs-2.5.93-1.fc26.x86_64
> and install gperftools-devel-2.5-2.fc25.x86_64 and
> gperftools-libs-2.5-2.fc25.x86_64 from
> http://download.bne.redhat.com/pub/fedora/linux/releases/25/Everything/x86_64/os/Packages/g/
> as I have had success up to this point with those packages (YMMV). Of course as
> Jeff pointed out you can also specify the libc allocator to cmake.
>
> Alternatively, I have created an unofficial package that contains the required
> patch here.
>
> https://copr-be.cloud.fedoraproject.org/results/badone/misc/fedora-26-x86_64/00606401-gperftools/
>
> So far this seems to work fine but it may cause problems so caveat emptor :)
>
> This patch should allow you to build with my package.
>
> diff --git a/cmake/modules/BuildRocksDB.cmake b/cmake/modules/BuildRocksDB.cmake
> index 8c44098d6f..efbd97fab2 100644
> --- a/cmake/modules/BuildRocksDB.cmake
> +++ b/cmake/modules/BuildRocksDB.cmake
> @@ -59,7 +59,8 @@ macro(build_rocksdb)
>      # see http://tracker.ceph.com/issues/21422
>      if(ROCKSDB_VERSION_STRING VERSION_GREATER 5.7 AND
>          TCMALLOC_VERSION_STRING VERSION_GREATER 2.5 AND
> -        TCMALLOC_VERSION_STRING VERSION_LESS 2.6.2)
> +        TCMALLOC_VERSION_STRING VERSION_LESS 2.6.2 AND NOT
> +        TCMALLOC_VERSION_STRING VERSION_EQUAL 2.5.93-x)
>        message(SEND_ERROR
>          "Incompatible tcmalloc v${TCMALLOC_VERSION_STRING} and rocksdb v${ROCKSDB_VERSION_STRING}, "
>          "please install gperf-tools 2.5 or > 2.6.2")
>
> I've also created a bug against fc26 here.
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1494309


thanks, Brad!

Patch
diff mbox

diff --git a/cmake/modules/BuildRocksDB.cmake b/cmake/modules/BuildRocksDB.cmake
index 8c44098d6f..efbd97fab2 100644
--- a/cmake/modules/BuildRocksDB.cmake
+++ b/cmake/modules/BuildRocksDB.cmake
@@ -59,7 +59,8 @@  macro(build_rocksdb)
     # see http://tracker.ceph.com/issues/21422
     if(ROCKSDB_VERSION_STRING VERSION_GREATER 5.7 AND
         TCMALLOC_VERSION_STRING VERSION_GREATER 2.5 AND
-        TCMALLOC_VERSION_STRING VERSION_LESS 2.6.2)
+        TCMALLOC_VERSION_STRING VERSION_LESS 2.6.2 AND NOT
+        TCMALLOC_VERSION_STRING VERSION_EQUAL 2.5.93-x)
       message(SEND_ERROR
         "Incompatible tcmalloc v${TCMALLOC_VERSION_STRING} and rocksdb v${ROCKSDB_VERSION_STRING}, "
         "please install gperf-tools 2.5 or > 2.6.2")