RocksDB Incorrect API Usage
diff mbox

Message ID CACJqLyYtHkeQ+jSNi7JPJzRb6wfCcNAuCSr_YyTwdqA_kzzP_g@mail.gmail.com
State New
Headers show

Commit Message

Haomai Wang May 31, 2016, 4:49 p.m. UTC
Hi Sage and Mark,

As mentioned in BlueStore standup, I found rocksdb iterator *Seek*
won't use bloom filter like *Get*.

*Get* impl: it will look at filter firstly
https://github.com/facebook/rocksdb/blob/master/table/block_based_table_reader.cc#L1369

Iterator *Seek*: it will do binary search, by default we don't specify
prefix feature(https://github.com/facebook/rocksdb/wiki/Prefix-Seek-API-Changes).
https://github.com/facebook/rocksdb/blob/master/table/block.cc#L94

So I use a simple tests:

./db_bench -num 10000000  -benchmarks fillbatch
fill the db firstly with 1000w records.

./db_bench -use_existing_db  -benchmarks readrandomfast
readrandomfast case will use *Get* API to retrive data

[root@hunter-node2 rocksdb]# ./db_bench -use_existing_db  -benchmarks
readrandomfast

LevelDB:    version 4.3
Date:       Wed Jun  1 00:29:16 2016
CPU:        32 * Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
CPUCache:   20480 KB
Keys:       16 bytes each
Values:     100 bytes each (50 bytes after compression)
Entries:    1000000
Prefix:    0 bytes
Keys per prefix:    0
RawSize:    110.6 MB (estimated)
FileSize:   62.9 MB (estimated)
Writes per second: 0
Compression: Snappy
Memtablerep: skip_list
Perf Level: 0
WARNING: Assertions are enabled; benchmarks unnecessarily slow
------------------------------------------------
DB path: [/tmp/rocksdbtest-0/dbbench]
readrandomfast :       4.570 micros/op 218806 ops/sec; (1000100 of
1000100 found, issued 46639 non-exist keys)

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Comments

Mark Nelson May 31, 2016, 4:56 p.m. UTC | #1
On 05/31/2016 11:49 AM, Haomai Wang wrote:
> Hi Sage and Mark,
>
> As mentioned in BlueStore standup, I found rocksdb iterator *Seek*
> won't use bloom filter like *Get*.
>
> *Get* impl: it will look at filter firstly
> https://github.com/facebook/rocksdb/blob/master/table/block_based_table_reader.cc#L1369
>
> Iterator *Seek*: it will do binary search, by default we don't specify
> prefix feature(https://github.com/facebook/rocksdb/wiki/Prefix-Seek-API-Changes).
> https://github.com/facebook/rocksdb/blob/master/table/block.cc#L94
>
> So I use a simple tests:
>
> ./db_bench -num 10000000  -benchmarks fillbatch
> fill the db firstly with 1000w records.
>
> ./db_bench -use_existing_db  -benchmarks readrandomfast
> readrandomfast case will use *Get* API to retrive data
>
> [root@hunter-node2 rocksdb]# ./db_bench -use_existing_db  -benchmarks
> readrandomfast
>
> LevelDB:    version 4.3
> Date:       Wed Jun  1 00:29:16 2016
> CPU:        32 * Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
> CPUCache:   20480 KB
> Keys:       16 bytes each
> Values:     100 bytes each (50 bytes after compression)
> Entries:    1000000
> Prefix:    0 bytes
> Keys per prefix:    0
> RawSize:    110.6 MB (estimated)
> FileSize:   62.9 MB (estimated)
> Writes per second: 0
> Compression: Snappy
> Memtablerep: skip_list
> Perf Level: 0
> WARNING: Assertions are enabled; benchmarks unnecessarily slow
> ------------------------------------------------
> DB path: [/tmp/rocksdbtest-0/dbbench]
> readrandomfast :       4.570 micros/op 218806 ops/sec; (1000100 of
> 1000100 found, issued 46639 non-exist keys)
>
> ===========================
> then I modify readrandomfast to use Iterator API[0]:
>
> [root@hunter-node2 rocksdb]# ./db_bench -use_existing_db  -benchmarks
> readrandomfast
> LevelDB:    version 4.3
> Date:       Wed Jun  1 00:33:03 2016
> CPU:        32 * Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
> CPUCache:   20480 KB
> Keys:       16 bytes each
> Values:     100 bytes each (50 bytes after compression)
> Entries:    1000000
> Prefix:    0 bytes
> Keys per prefix:    0
> RawSize:    110.6 MB (estimated)
> FileSize:   62.9 MB (estimated)
> Writes per second: 0
> Compression: Snappy
> Memtablerep: skip_list
> Perf Level: 0
> WARNING: Assertions are enabled; benchmarks unnecessarily slow
> ------------------------------------------------
> DB path: [/tmp/rocksdbtest-0/dbbench]
> readrandomfast :      45.188 micros/op 22129 ops/sec; (1000100 of
> 1000100 found, issued 46639 non-exist keys)
>
>
> 45.18 us/op vs 4.57us/op!
>
> The test can be repeated and easy to do! Plz correct if I'm doing
> foolish thing I'm not aware..

Excellent catch Haomai!  I'm not sure I will be able to test before I 
leave on holiday, but if I do I will report back.  Do you think upstream 
rocksdb can be improved to make the iterator implementation faster?

>
> So I proposal this PR: https://github.com/ceph/ceph/pull/9411
>
> We still can make further improvements by scanning all iterate usage
> to make it better!
>
> [0]:
> --- a/db/db_bench.cc
> +++ b/db/db_bench.cc
> @@ -2923,14 +2923,12 @@ class Benchmark {
>          int64_t key_rand = thread->rand.Next() & (pot - 1);
>          GenerateKeyFromInt(key_rand, FLAGS_num, &key);
>          ++read;
> -        auto status = db->Get(options, key, &value);
> -        if (status.ok()) {
> -          ++found;
> -        } else if (!status.IsNotFound()) {
> -          fprintf(stderr, "Get returned an error: %s\n",
> -                  status.ToString().c_str());
> -          abort();
> -        }
> +        Iterator* iter = db->NewIterator(options);
> +      iter->Seek(key);
> +      if (iter->Valid() && iter->key().compare(key) == 0) {
> +        found++;
> +      }
> +
>          if (key_rand >= FLAGS_num) {
>            ++nonexist;
>          }
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Piotr Dałek May 31, 2016, 5:17 p.m. UTC | #2
On Wed, Jun 01, 2016 at 12:49:53AM +0800, Haomai Wang wrote:
> Hi Sage and Mark,
> 
> As mentioned in BlueStore standup, I found rocksdb iterator *Seek*
> won't use bloom filter like *Get*.
> 
> *Get* impl: it will look at filter firstly
> https://github.com/facebook/rocksdb/blob/master/table/block_based_table_reader.cc#L1369
> 
> Iterator *Seek*: it will do binary search, by default we don't specify
> prefix feature(https://github.com/facebook/rocksdb/wiki/Prefix-Seek-API-Changes).
> https://github.com/facebook/rocksdb/blob/master/table/block.cc#L94
> 
> [..]
> --- a/db/db_bench.cc
> +++ b/db/db_bench.cc
> @@ -2923,14 +2923,12 @@ class Benchmark {
>          int64_t key_rand = thread->rand.Next() & (pot - 1);
>          GenerateKeyFromInt(key_rand, FLAGS_num, &key);
>          ++read;
> -        auto status = db->Get(options, key, &value);
> -        if (status.ok()) {
> -          ++found;
> -        } else if (!status.IsNotFound()) {
> -          fprintf(stderr, "Get returned an error: %s\n",
> -                  status.ToString().c_str());
> -          abort();
> -        }
> +        Iterator* iter = db->NewIterator(options);
> +      iter->Seek(key);
> +      if (iter->Valid() && iter->key().compare(key) == 0) {
> +        found++;
> +      }
> +
>          if (key_rand >= FLAGS_num) {
>            ++nonexist;
>          }

Aren't you missing "delete iter" here?
Haomai Wang May 31, 2016, 5:47 p.m. UTC | #3
On Wed, Jun 1, 2016 at 1:17 AM, Piotr Dałek <branch@predictor.org.pl> wrote:
> On Wed, Jun 01, 2016 at 12:49:53AM +0800, Haomai Wang wrote:
>> Hi Sage and Mark,
>>
>> As mentioned in BlueStore standup, I found rocksdb iterator *Seek*
>> won't use bloom filter like *Get*.
>>
>> *Get* impl: it will look at filter firstly
>> https://github.com/facebook/rocksdb/blob/master/table/block_based_table_reader.cc#L1369
>>
>> Iterator *Seek*: it will do binary search, by default we don't specify
>> prefix feature(https://github.com/facebook/rocksdb/wiki/Prefix-Seek-API-Changes).
>> https://github.com/facebook/rocksdb/blob/master/table/block.cc#L94
>>
>> [..]
>> --- a/db/db_bench.cc
>> +++ b/db/db_bench.cc
>> @@ -2923,14 +2923,12 @@ class Benchmark {
>>          int64_t key_rand = thread->rand.Next() & (pot - 1);
>>          GenerateKeyFromInt(key_rand, FLAGS_num, &key);
>>          ++read;
>> -        auto status = db->Get(options, key, &value);
>> -        if (status.ok()) {
>> -          ++found;
>> -        } else if (!status.IsNotFound()) {
>> -          fprintf(stderr, "Get returned an error: %s\n",
>> -                  status.ToString().c_str());
>> -          abort();
>> -        }
>> +        Iterator* iter = db->NewIterator(options);
>> +      iter->Seek(key);
>> +      if (iter->Valid() && iter->key().compare(key) == 0) {
>> +        found++;
>> +      }
>> +
>>          if (key_rand >= FLAGS_num) {
>>            ++nonexist;
>>          }
>
> Aren't you missing "delete iter" here?

oh, sure. I retest this, now it's 20us/op. Much better :-). 20 vs 4.

And actually this test set is seq int key and the value is small, it
will be make the gap not so much. Consider the binary search vs bloom
filter, I think the latency is huge. We shouldn't use any iterate
except we ensure it's a continuous key set.

>
> --
> Piotr Dałek
> branch@predictor.org.pl
> http://blog.predictor.org.pl
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Jianjian Huo May 31, 2016, 6:08 p.m. UTC | #4
Hi Haomai,

I noticed this as well, and made same changes to RocksDBStore in this PR last week:
https://github.com/ceph/ceph/pull/9215

One thing which is even worse,  seek will bypass row cache, so kv pairs won't be able to be cached in row cache.
I am working to benchmark the performance impact, will publish the results after I am done this week.

Jianjian

On Tue, May 31, 2016 at 9:49 AM, Haomai Wang <haomai@xsky.com> wrote:
> Hi Sage and Mark,

>

> As mentioned in BlueStore standup, I found rocksdb iterator *Seek*

> won't use bloom filter like *Get*.

>

> *Get* impl: it will look at filter firstly

> https://github.com/facebook/rocksdb/blob/master/table/block_based_table_reader.cc#L1369

>

> Iterator *Seek*: it will do binary search, by default we don't specify

> prefix feature(https://github.com/facebook/rocksdb/wiki/Prefix-Seek-API-Changes).

> https://github.com/facebook/rocksdb/blob/master/table/block.cc#L94

>

> So I use a simple tests:

>

> ./db_bench -num 10000000  -benchmarks fillbatch

> fill the db firstly with 1000w records.

>

> ./db_bench -use_existing_db  -benchmarks readrandomfast

> readrandomfast case will use *Get* API to retrive data

>

> [root@hunter-node2 rocksdb]# ./db_bench -use_existing_db  -benchmarks

> readrandomfast

>

> LevelDB:    version 4.3

> Date:       Wed Jun  1 00:29:16 2016

> CPU:        32 * Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz

> CPUCache:   20480 KB

> Keys:       16 bytes each

> Values:     100 bytes each (50 bytes after compression)

> Entries:    1000000

> Prefix:    0 bytes

> Keys per prefix:    0

> RawSize:    110.6 MB (estimated)

> FileSize:   62.9 MB (estimated)

> Writes per second: 0

> Compression: Snappy

> Memtablerep: skip_list

> Perf Level: 0

> WARNING: Assertions are enabled; benchmarks unnecessarily slow

> ------------------------------------------------

> DB path: [/tmp/rocksdbtest-0/dbbench]

> readrandomfast :       4.570 micros/op 218806 ops/sec; (1000100 of

> 1000100 found, issued 46639 non-exist keys)

>

> ===========================

> then I modify readrandomfast to use Iterator API[0]:

>

> [root@hunter-node2 rocksdb]# ./db_bench -use_existing_db  -benchmarks

> readrandomfast

> LevelDB:    version 4.3

> Date:       Wed Jun  1 00:33:03 2016

> CPU:        32 * Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz

> CPUCache:   20480 KB

> Keys:       16 bytes each

> Values:     100 bytes each (50 bytes after compression)

> Entries:    1000000

> Prefix:    0 bytes

> Keys per prefix:    0

> RawSize:    110.6 MB (estimated)

> FileSize:   62.9 MB (estimated)

> Writes per second: 0

> Compression: Snappy

> Memtablerep: skip_list

> Perf Level: 0

> WARNING: Assertions are enabled; benchmarks unnecessarily slow

> ------------------------------------------------

> DB path: [/tmp/rocksdbtest-0/dbbench]

> readrandomfast :      45.188 micros/op 22129 ops/sec; (1000100 of

> 1000100 found, issued 46639 non-exist keys)

>

>

> 45.18 us/op vs 4.57us/op!

>

> The test can be repeated and easy to do! Plz correct if I'm doing

> foolish thing I'm not aware..

>

> So I proposal this PR: https://github.com/ceph/ceph/pull/9411

>

> We still can make further improvements by scanning all iterate usage

> to make it better!

>

> [0]:

> --- a/db/db_bench.cc

> +++ b/db/db_bench.cc

> @@ -2923,14 +2923,12 @@ class Benchmark {

>          int64_t key_rand = thread->rand.Next() & (pot - 1);

>          GenerateKeyFromInt(key_rand, FLAGS_num, &key);

>          ++read;

> -        auto status = db->Get(options, key, &value);

> -        if (status.ok()) {

> -          ++found;

> -        } else if (!status.IsNotFound()) {

> -          fprintf(stderr, "Get returned an error: %s\n",

> -                  status.ToString().c_str());

> -          abort();

> -        }

> +        Iterator* iter = db->NewIterator(options);

> +      iter->Seek(key);

> +      if (iter->Valid() && iter->key().compare(key) == 0) {

> +        found++;

> +      }

> +

>          if (key_rand >= FLAGS_num) {

>            ++nonexist;

>          }

> --

> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in

> the body of a message to majordomo@vger.kernel.org

> More majordomo info at  http://vger.kernel.org/majordomo-info.html
Haomai Wang May 31, 2016, 6:12 p.m. UTC | #5
On Wed, Jun 1, 2016 at 2:08 AM, Jianjian Huo <jianjian.huo@samsung.com> wrote:
> Hi Haomai,
>
> I noticed this as well, and made same changes to RocksDBStore in this PR last week:
> https://github.com/ceph/ceph/pull/9215
>
> One thing which is even worse,  seek will bypass row cache, so kv pairs won't be able to be cached in row cache.
> I am working to benchmark the performance impact, will publish the results after I am done this week.

Oh, cool! I think you can cherry-pick my another leveldb fix.

BTW, do you pay an attention to prefix seek api? I think it will be
more suitable than column family in ceph case. If we can have
well-defined prefix rule, we can make most of range query cheaper!

>
> Jianjian
>
> On Tue, May 31, 2016 at 9:49 AM, Haomai Wang <haomai@xsky.com> wrote:
>> Hi Sage and Mark,
>>
>> As mentioned in BlueStore standup, I found rocksdb iterator *Seek*
>> won't use bloom filter like *Get*.
>>
>> *Get* impl: it will look at filter firstly
>> https://github.com/facebook/rocksdb/blob/master/table/block_based_table_reader.cc#L1369
>>
>> Iterator *Seek*: it will do binary search, by default we don't specify
>> prefix feature(https://github.com/facebook/rocksdb/wiki/Prefix-Seek-API-Changes).
>> https://github.com/facebook/rocksdb/blob/master/table/block.cc#L94
>>
>> So I use a simple tests:
>>
>> ./db_bench -num 10000000  -benchmarks fillbatch
>> fill the db firstly with 1000w records.
>>
>> ./db_bench -use_existing_db  -benchmarks readrandomfast
>> readrandomfast case will use *Get* API to retrive data
>>
>> [root@hunter-node2 rocksdb]# ./db_bench -use_existing_db  -benchmarks
>> readrandomfast
>>
>> LevelDB:    version 4.3
>> Date:       Wed Jun  1 00:29:16 2016
>> CPU:        32 * Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
>> CPUCache:   20480 KB
>> Keys:       16 bytes each
>> Values:     100 bytes each (50 bytes after compression)
>> Entries:    1000000
>> Prefix:    0 bytes
>> Keys per prefix:    0
>> RawSize:    110.6 MB (estimated)
>> FileSize:   62.9 MB (estimated)
>> Writes per second: 0
>> Compression: Snappy
>> Memtablerep: skip_list
>> Perf Level: 0
>> WARNING: Assertions are enabled; benchmarks unnecessarily slow
>> ------------------------------------------------
>> DB path: [/tmp/rocksdbtest-0/dbbench]
>> readrandomfast :       4.570 micros/op 218806 ops/sec; (1000100 of
>> 1000100 found, issued 46639 non-exist keys)
>>
>> ===========================
>> then I modify readrandomfast to use Iterator API[0]:
>>
>> [root@hunter-node2 rocksdb]# ./db_bench -use_existing_db  -benchmarks
>> readrandomfast
>> LevelDB:    version 4.3
>> Date:       Wed Jun  1 00:33:03 2016
>> CPU:        32 * Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
>> CPUCache:   20480 KB
>> Keys:       16 bytes each
>> Values:     100 bytes each (50 bytes after compression)
>> Entries:    1000000
>> Prefix:    0 bytes
>> Keys per prefix:    0
>> RawSize:    110.6 MB (estimated)
>> FileSize:   62.9 MB (estimated)
>> Writes per second: 0
>> Compression: Snappy
>> Memtablerep: skip_list
>> Perf Level: 0
>> WARNING: Assertions are enabled; benchmarks unnecessarily slow
>> ------------------------------------------------
>> DB path: [/tmp/rocksdbtest-0/dbbench]
>> readrandomfast :      45.188 micros/op 22129 ops/sec; (1000100 of
>> 1000100 found, issued 46639 non-exist keys)
>>
>>
>> 45.18 us/op vs 4.57us/op!
>>
>> The test can be repeated and easy to do! Plz correct if I'm doing
>> foolish thing I'm not aware..
>>
>> So I proposal this PR: https://github.com/ceph/ceph/pull/9411
>>
>> We still can make further improvements by scanning all iterate usage
>> to make it better!
>>
>> [0]:
>> --- a/db/db_bench.cc
>> +++ b/db/db_bench.cc
>> @@ -2923,14 +2923,12 @@ class Benchmark {
>>          int64_t key_rand = thread->rand.Next() & (pot - 1);
>>          GenerateKeyFromInt(key_rand, FLAGS_num, &key);
>>          ++read;
>> -        auto status = db->Get(options, key, &value);
>> -        if (status.ok()) {
>> -          ++found;
>> -        } else if (!status.IsNotFound()) {
>> -          fprintf(stderr, "Get returned an error: %s\n",
>> -                  status.ToString().c_str());
>> -          abort();
>> -        }
>> +        Iterator* iter = db->NewIterator(options);
>> +      iter->Seek(key);
>> +      if (iter->Valid() && iter->key().compare(key) == 0) {
>> +        found++;
>> +      }
>> +
>>          if (key_rand >= FLAGS_num) {
>>            ++nonexist;
>>          }
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Jianjian Huo May 31, 2016, 7:40 p.m. UTC | #6
On Tue, May 31, 2016 at 11:12 AM, Haomai Wang <haomai@xsky.com> wrote:
> On Wed, Jun 1, 2016 at 2:08 AM, Jianjian Huo <jianjian.huo@samsung.com> wrote:

>> Hi Haomai,

>>

>> I noticed this as well, and made same changes to RocksDBStore in this PR last week:

>> https://github.com/ceph/ceph/pull/9215

>>

>> One thing which is even worse,  seek will bypass row cache, so kv pairs won't be able to be cached in row cache.

>> I am working to benchmark the performance impact, will publish the results after I am done this week.

>

> Oh, cool! I think you can cherry-pick my another leveldb fix.


No problem, I will do that. 
>

> BTW, do you pay an attention to prefix seek api? I think it will be

> more suitable than column family in ceph case. If we can have

> well-defined prefix rule, we can make most of range query cheaper!


When keys with same prefix are stored in their own SST files(CF case), even seeking without prefix will be faster than seeking with prefix but mixed with different prefixed keys?
I am not sure what optimization prefix seek will use internally for block based format, but to me, it's hard to beat the case when you only have one prefixed keys stored separately.
>

>>

>> Jianjian

>>

>> On Tue, May 31, 2016 at 9:49 AM, Haomai Wang <haomai@xsky.com> wrote:

>>> Hi Sage and Mark,

>>>

>>> As mentioned in BlueStore standup, I found rocksdb iterator *Seek*

>>> won't use bloom filter like *Get*.

>>>

>>> *Get* impl: it will look at filter firstly

>>> https://github.com/facebook/rocksdb/blob/master/table/block_based_table_reader.cc#L1369

>>>

>>> Iterator *Seek*: it will do binary search, by default we don't specify

>>> prefix feature(https://github.com/facebook/rocksdb/wiki/Prefix-Seek-API-Changes).

>>> https://github.com/facebook/rocksdb/blob/master/table/block.cc#L94

>>>

>>> So I use a simple tests:

>>>

>>> ./db_bench -num 10000000  -benchmarks fillbatch

>>> fill the db firstly with 1000w records.

>>>

>>> ./db_bench -use_existing_db  -benchmarks readrandomfast

>>> readrandomfast case will use *Get* API to retrive data

>>>

>>> [root@hunter-node2 rocksdb]# ./db_bench -use_existing_db  -benchmarks

>>> readrandomfast

>>>

>>> LevelDB:    version 4.3

>>> Date:       Wed Jun  1 00:29:16 2016

>>> CPU:        32 * Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz

>>> CPUCache:   20480 KB

>>> Keys:       16 bytes each

>>> Values:     100 bytes each (50 bytes after compression)

>>> Entries:    1000000

>>> Prefix:    0 bytes

>>> Keys per prefix:    0

>>> RawSize:    110.6 MB (estimated)

>>> FileSize:   62.9 MB (estimated)

>>> Writes per second: 0

>>> Compression: Snappy

>>> Memtablerep: skip_list

>>> Perf Level: 0

>>> WARNING: Assertions are enabled; benchmarks unnecessarily slow

>>> ------------------------------------------------

>>> DB path: [/tmp/rocksdbtest-0/dbbench]

>>> readrandomfast :       4.570 micros/op 218806 ops/sec; (1000100 of

>>> 1000100 found, issued 46639 non-exist keys)

>>>

>>> ===========================

>>> then I modify readrandomfast to use Iterator API[0]:

>>>

>>> [root@hunter-node2 rocksdb]# ./db_bench -use_existing_db  -benchmarks

>>> readrandomfast

>>> LevelDB:    version 4.3

>>> Date:       Wed Jun  1 00:33:03 2016

>>> CPU:        32 * Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz

>>> CPUCache:   20480 KB

>>> Keys:       16 bytes each

>>> Values:     100 bytes each (50 bytes after compression)

>>> Entries:    1000000

>>> Prefix:    0 bytes

>>> Keys per prefix:    0

>>> RawSize:    110.6 MB (estimated)

>>> FileSize:   62.9 MB (estimated)

>>> Writes per second: 0

>>> Compression: Snappy

>>> Memtablerep: skip_list

>>> Perf Level: 0

>>> WARNING: Assertions are enabled; benchmarks unnecessarily slow

>>> ------------------------------------------------

>>> DB path: [/tmp/rocksdbtest-0/dbbench]

>>> readrandomfast :      45.188 micros/op 22129 ops/sec; (1000100 of

>>> 1000100 found, issued 46639 non-exist keys)

>>>

>>>

>>> 45.18 us/op vs 4.57us/op!

>>>

>>> The test can be repeated and easy to do! Plz correct if I'm doing

>>> foolish thing I'm not aware..

>>>

>>> So I proposal this PR: https://github.com/ceph/ceph/pull/9411

>>>

>>> We still can make further improvements by scanning all iterate usage

>>> to make it better!

>>>

>>> [0]:

>>> --- a/db/db_bench.cc

>>> +++ b/db/db_bench.cc

>>> @@ -2923,14 +2923,12 @@ class Benchmark {

>>>          int64_t key_rand = thread->rand.Next() & (pot - 1);

>>>          GenerateKeyFromInt(key_rand, FLAGS_num, &key);

>>>          ++read;

>>> -        auto status = db->Get(options, key, &value);

>>> -        if (status.ok()) {

>>> -          ++found;

>>> -        } else if (!status.IsNotFound()) {

>>> -          fprintf(stderr, "Get returned an error: %s\n",

>>> -                  status.ToString().c_str());

>>> -          abort();

>>> -        }

>>> +        Iterator* iter = db->NewIterator(options);

>>> +      iter->Seek(key);

>>> +      if (iter->Valid() && iter->key().compare(key) == 0) {

>>> +        found++;

>>> +      }

>>> +

>>>          if (key_rand >= FLAGS_num) {

>>>            ++nonexist;

>>>          }

>>> --

>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in

>>> the body of a message to majordomo@vger.kernel.org

>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

> --

> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in

> the body of a message to majordomo@vger.kernel.org

> More majordomo info at  http://vger.kernel.org/majordomo-info.html
Haomai Wang June 1, 2016, 2:38 a.m. UTC | #7
On Wed, Jun 1, 2016 at 3:40 AM, Jianjian Huo <jianjian.huo@samsung.com> wrote:
>
> On Tue, May 31, 2016 at 11:12 AM, Haomai Wang <haomai@xsky.com> wrote:
>> On Wed, Jun 1, 2016 at 2:08 AM, Jianjian Huo <jianjian.huo@samsung.com> wrote:
>>> Hi Haomai,
>>>
>>> I noticed this as well, and made same changes to RocksDBStore in this PR last week:
>>> https://github.com/ceph/ceph/pull/9215
>>>
>>> One thing which is even worse,  seek will bypass row cache, so kv pairs won't be able to be cached in row cache.
>>> I am working to benchmark the performance impact, will publish the results after I am done this week.
>>
>> Oh, cool! I think you can cherry-pick my another leveldb fix.
>
> No problem, I will do that.
>>
>> BTW, do you pay an attention to prefix seek api? I think it will be
>> more suitable than column family in ceph case. If we can have
>> well-defined prefix rule, we can make most of range query cheaper!
>
> When keys with same prefix are stored in their own SST files(CF case), even seeking without prefix will be faster than seeking with prefix but mixed with different prefixed keys?

From my current view, prefix could be more flexible. For example, each
rgw bucket index could use one prefix to make each bucket index object
seek separated. For CF, it would be too heavry.

> I am not sure what optimization prefix seek will use internally for block based format, but to me, it's hard to beat the case when you only have one prefixed keys stored separately.
>>
>>>
>>> Jianjian
>>>
>>> On Tue, May 31, 2016 at 9:49 AM, Haomai Wang <haomai@xsky.com> wrote:
>>>> Hi Sage and Mark,
>>>>
>>>> As mentioned in BlueStore standup, I found rocksdb iterator *Seek*
>>>> won't use bloom filter like *Get*.
>>>>
>>>> *Get* impl: it will look at filter firstly
>>>> https://github.com/facebook/rocksdb/blob/master/table/block_based_table_reader.cc#L1369
>>>>
>>>> Iterator *Seek*: it will do binary search, by default we don't specify
>>>> prefix feature(https://github.com/facebook/rocksdb/wiki/Prefix-Seek-API-Changes).
>>>> https://github.com/facebook/rocksdb/blob/master/table/block.cc#L94
>>>>
>>>> So I use a simple tests:
>>>>
>>>> ./db_bench -num 10000000  -benchmarks fillbatch
>>>> fill the db firstly with 1000w records.
>>>>
>>>> ./db_bench -use_existing_db  -benchmarks readrandomfast
>>>> readrandomfast case will use *Get* API to retrive data
>>>>
>>>> [root@hunter-node2 rocksdb]# ./db_bench -use_existing_db  -benchmarks
>>>> readrandomfast
>>>>
>>>> LevelDB:    version 4.3
>>>> Date:       Wed Jun  1 00:29:16 2016
>>>> CPU:        32 * Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
>>>> CPUCache:   20480 KB
>>>> Keys:       16 bytes each
>>>> Values:     100 bytes each (50 bytes after compression)
>>>> Entries:    1000000
>>>> Prefix:    0 bytes
>>>> Keys per prefix:    0
>>>> RawSize:    110.6 MB (estimated)
>>>> FileSize:   62.9 MB (estimated)
>>>> Writes per second: 0
>>>> Compression: Snappy
>>>> Memtablerep: skip_list
>>>> Perf Level: 0
>>>> WARNING: Assertions are enabled; benchmarks unnecessarily slow
>>>> ------------------------------------------------
>>>> DB path: [/tmp/rocksdbtest-0/dbbench]
>>>> readrandomfast :       4.570 micros/op 218806 ops/sec; (1000100 of
>>>> 1000100 found, issued 46639 non-exist keys)
>>>>
>>>> ===========================
>>>> then I modify readrandomfast to use Iterator API[0]:
>>>>
>>>> [root@hunter-node2 rocksdb]# ./db_bench -use_existing_db  -benchmarks
>>>> readrandomfast
>>>> LevelDB:    version 4.3
>>>> Date:       Wed Jun  1 00:33:03 2016
>>>> CPU:        32 * Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
>>>> CPUCache:   20480 KB
>>>> Keys:       16 bytes each
>>>> Values:     100 bytes each (50 bytes after compression)
>>>> Entries:    1000000
>>>> Prefix:    0 bytes
>>>> Keys per prefix:    0
>>>> RawSize:    110.6 MB (estimated)
>>>> FileSize:   62.9 MB (estimated)
>>>> Writes per second: 0
>>>> Compression: Snappy
>>>> Memtablerep: skip_list
>>>> Perf Level: 0
>>>> WARNING: Assertions are enabled; benchmarks unnecessarily slow
>>>> ------------------------------------------------
>>>> DB path: [/tmp/rocksdbtest-0/dbbench]
>>>> readrandomfast :      45.188 micros/op 22129 ops/sec; (1000100 of
>>>> 1000100 found, issued 46639 non-exist keys)
>>>>
>>>>
>>>> 45.18 us/op vs 4.57us/op!
>>>>
>>>> The test can be repeated and easy to do! Plz correct if I'm doing
>>>> foolish thing I'm not aware..
>>>>
>>>> So I proposal this PR: https://github.com/ceph/ceph/pull/9411
>>>>
>>>> We still can make further improvements by scanning all iterate usage
>>>> to make it better!
>>>>
>>>> [0]:
>>>> --- a/db/db_bench.cc
>>>> +++ b/db/db_bench.cc
>>>> @@ -2923,14 +2923,12 @@ class Benchmark {
>>>>          int64_t key_rand = thread->rand.Next() & (pot - 1);
>>>>          GenerateKeyFromInt(key_rand, FLAGS_num, &key);
>>>>          ++read;
>>>> -        auto status = db->Get(options, key, &value);
>>>> -        if (status.ok()) {
>>>> -          ++found;
>>>> -        } else if (!status.IsNotFound()) {
>>>> -          fprintf(stderr, "Get returned an error: %s\n",
>>>> -                  status.ToString().c_str());
>>>> -          abort();
>>>> -        }
>>>> +        Iterator* iter = db->NewIterator(options);
>>>> +      iter->Seek(key);
>>>> +      if (iter->Valid() && iter->key().compare(key) == 0) {
>>>> +        found++;
>>>> +      }
>>>> +
>>>>          if (key_rand >= FLAGS_num) {
>>>>            ++nonexist;
>>>>          }
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Allen Samuels June 1, 2016, 2:52 a.m. UTC | #8
PiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBjZXBoLWRldmVsLW93bmVyQHZn
ZXIua2VybmVsLm9yZyBbbWFpbHRvOmNlcGgtZGV2ZWwtDQo+IG93bmVyQHZnZXIua2VybmVsLm9y
Z10gT24gQmVoYWxmIE9mIEhhb21haSBXYW5nDQo+IFNlbnQ6IFR1ZXNkYXksIE1heSAzMSwgMjAx
NiA3OjM4IFBNDQo+IFRvOiBKaWFuamlhbiBIdW8gPGppYW5qaWFuLmh1b0BzYW1zdW5nLmNvbT4N
Cj4gQ2M6IFNhZ2UgV2VpbCA8c3dlaWxAcmVkaGF0LmNvbT47IE1hcmsgTmVsc29uIDxtbmVsc29u
QHJlZGhhdC5jb20+Ow0KPiBjZXBoLWRldmVsQHZnZXIua2VybmVsLm9yZw0KPiBTdWJqZWN0OiBS
ZTogUm9ja3NEQiBJbmNvcnJlY3QgQVBJIFVzYWdlDQo+IA0KPiBPbiBXZWQsIEp1biAxLCAyMDE2
IGF0IDM6NDAgQU0sIEppYW5qaWFuIEh1byA8amlhbmppYW4uaHVvQHNhbXN1bmcuY29tPg0KPiB3
cm90ZToNCj4gPg0KPiA+IE9uIFR1ZSwgTWF5IDMxLCAyMDE2IGF0IDExOjEyIEFNLCBIYW9tYWkg
V2FuZyA8aGFvbWFpQHhza3kuY29tPg0KPiB3cm90ZToNCj4gPj4gT24gV2VkLCBKdW4gMSwgMjAx
NiBhdCAyOjA4IEFNLCBKaWFuamlhbiBIdW8NCj4gPGppYW5qaWFuLmh1b0BzYW1zdW5nLmNvbT4g
d3JvdGU6DQo+ID4+PiBIaSBIYW9tYWksDQo+ID4+Pg0KPiA+Pj4gSSBub3RpY2VkIHRoaXMgYXMg
d2VsbCwgYW5kIG1hZGUgc2FtZSBjaGFuZ2VzIHRvIFJvY2tzREJTdG9yZSBpbiB0aGlzIFBSDQo+
IGxhc3Qgd2VlazoNCj4gPj4+IGh0dHBzOi8vZ2l0aHViLmNvbS9jZXBoL2NlcGgvcHVsbC85MjE1
DQo+ID4+Pg0KPiA+Pj4gT25lIHRoaW5nIHdoaWNoIGlzIGV2ZW4gd29yc2UsICBzZWVrIHdpbGwg
YnlwYXNzIHJvdyBjYWNoZSwgc28ga3YgcGFpcnMNCj4gd29uJ3QgYmUgYWJsZSB0byBiZSBjYWNo
ZWQgaW4gcm93IGNhY2hlLg0KPiA+Pj4gSSBhbSB3b3JraW5nIHRvIGJlbmNobWFyayB0aGUgcGVy
Zm9ybWFuY2UgaW1wYWN0LCB3aWxsIHB1Ymxpc2ggdGhlDQo+IHJlc3VsdHMgYWZ0ZXIgSSBhbSBk
b25lIHRoaXMgd2Vlay4NCj4gPj4NCj4gPj4gT2gsIGNvb2whIEkgdGhpbmsgeW91IGNhbiBjaGVy
cnktcGljayBteSBhbm90aGVyIGxldmVsZGIgZml4Lg0KPiA+DQo+ID4gTm8gcHJvYmxlbSwgSSB3
aWxsIGRvIHRoYXQuDQo+ID4+DQo+ID4+IEJUVywgZG8geW91IHBheSBhbiBhdHRlbnRpb24gdG8g
cHJlZml4IHNlZWsgYXBpPyBJIHRoaW5rIGl0IHdpbGwgYmUNCj4gPj4gbW9yZSBzdWl0YWJsZSB0
aGFuIGNvbHVtbiBmYW1pbHkgaW4gY2VwaCBjYXNlLiBJZiB3ZSBjYW4gaGF2ZQ0KPiA+PiB3ZWxs
LWRlZmluZWQgcHJlZml4IHJ1bGUsIHdlIGNhbiBtYWtlIG1vc3Qgb2YgcmFuZ2UgcXVlcnkgY2hl
YXBlciENCj4gPg0KPiA+IFdoZW4ga2V5cyB3aXRoIHNhbWUgcHJlZml4IGFyZSBzdG9yZWQgaW4g
dGhlaXIgb3duIFNTVCBmaWxlcyhDRiBjYXNlKSwNCj4gZXZlbiBzZWVraW5nIHdpdGhvdXQgcHJl
Zml4IHdpbGwgYmUgZmFzdGVyIHRoYW4gc2Vla2luZyB3aXRoIHByZWZpeCBidXQgbWl4ZWQNCj4g
d2l0aCBkaWZmZXJlbnQgcHJlZml4ZWQga2V5cz8NCj4gDQo+IEZyb20gbXkgY3VycmVudCB2aWV3
LCBwcmVmaXggY291bGQgYmUgbW9yZSBmbGV4aWJsZS4gRm9yIGV4YW1wbGUsIGVhY2ggcmd3DQo+
IGJ1Y2tldCBpbmRleCBjb3VsZCB1c2Ugb25lIHByZWZpeCB0byBtYWtlIGVhY2ggYnVja2V0IGlu
ZGV4IG9iamVjdCBzZWVrDQo+IHNlcGFyYXRlZC4gRm9yIENGLCBpdCB3b3VsZCBiZSB0b28gaGVh
dnJ5Lg0KDQpUaGlzIG1pZ2h0IGNhdXNlIG90aGVyIHByb2JsZW1zLiBXb3VsZCBpdCBkcmFtYXRp
Y2FsbHkgaW5jcmVhc2UgdGhlIG51bWJlciBvZiBmaWxlcyB0aGF0IEJsdWVGUyBuZWVkcyB0byBt
YW5hZ2U/IElmIHNvLCB0aGF0IG1pZ2h0IGVmZmVjdGl2ZWx5IGJyZWFrIHRoYXQgY29kZSB0b28g
KG9mIGNvdXJzZSBpdCdzIGZpeGFibGUgYWxzbyA6KSkNCj4gDQo+ID4gSSBhbSBub3Qgc3VyZSB3
aGF0IG9wdGltaXphdGlvbiBwcmVmaXggc2VlayB3aWxsIHVzZSBpbnRlcm5hbGx5IGZvciBibG9j
aw0KPiBiYXNlZCBmb3JtYXQsIGJ1dCB0byBtZSwgaXQncyBoYXJkIHRvIGJlYXQgdGhlIGNhc2Ug
d2hlbiB5b3Ugb25seSBoYXZlIG9uZQ0KPiBwcmVmaXhlZCBrZXlzIHN0b3JlZCBzZXBhcmF0ZWx5
Lg0KPiA+Pg0KPiA+Pj4NCj4gPj4+IEppYW5qaWFuDQo+ID4+Pg0KPiA+Pj4gT24gVHVlLCBNYXkg
MzEsIDIwMTYgYXQgOTo0OSBBTSwgSGFvbWFpIFdhbmcgPGhhb21haUB4c2t5LmNvbT4NCj4gd3Jv
dGU6DQo+ID4+Pj4gSGkgU2FnZSBhbmQgTWFyaywNCj4gPj4+Pg0KPiA+Pj4+IEFzIG1lbnRpb25l
ZCBpbiBCbHVlU3RvcmUgc3RhbmR1cCwgSSBmb3VuZCByb2Nrc2RiIGl0ZXJhdG9yICpTZWVrKg0K
PiA+Pj4+IHdvbid0IHVzZSBibG9vbSBmaWx0ZXIgbGlrZSAqR2V0Ki4NCj4gPj4+Pg0KPiA+Pj4+
ICpHZXQqIGltcGw6IGl0IHdpbGwgbG9vayBhdCBmaWx0ZXIgZmlyc3RseQ0KPiA+Pj4+DQo+IGh0
dHBzOi8vZ2l0aHViLmNvbS9mYWNlYm9vay9yb2Nrc2RiL2Jsb2IvbWFzdGVyL3RhYmxlL2Jsb2Nr
X2Jhc2VkX3QNCj4gPj4+PiBhYmxlX3JlYWRlci5jYyNMMTM2OQ0KPiA+Pj4+DQo+ID4+Pj4gSXRl
cmF0b3IgKlNlZWsqOiBpdCB3aWxsIGRvIGJpbmFyeSBzZWFyY2gsIGJ5IGRlZmF1bHQgd2UgZG9u
J3QNCj4gPj4+PiBzcGVjaWZ5IHByZWZpeA0KPiBmZWF0dXJlKGh0dHBzOi8vZ2l0aHViLmNvbS9m
YWNlYm9vay9yb2Nrc2RiL3dpa2kvUHJlZml4LVNlZWstQVBJLQ0KPiBDaGFuZ2VzKS4NCj4gPj4+
PiBodHRwczovL2dpdGh1Yi5jb20vZmFjZWJvb2svcm9ja3NkYi9ibG9iL21hc3Rlci90YWJsZS9i
bG9jay5jYyNMOTQNCj4gPj4+Pg0KPiA+Pj4+IFNvIEkgdXNlIGEgc2ltcGxlIHRlc3RzOg0KPiA+
Pj4+DQo+ID4+Pj4gLi9kYl9iZW5jaCAtbnVtIDEwMDAwMDAwICAtYmVuY2htYXJrcyBmaWxsYmF0
Y2ggZmlsbCB0aGUgZGIgZmlyc3RseQ0KPiA+Pj4+IHdpdGggMTAwMHcgcmVjb3Jkcy4NCj4gPj4+
Pg0KPiA+Pj4+IC4vZGJfYmVuY2ggLXVzZV9leGlzdGluZ19kYiAgLWJlbmNobWFya3MgcmVhZHJh
bmRvbWZhc3QNCj4gPj4+PiByZWFkcmFuZG9tZmFzdCBjYXNlIHdpbGwgdXNlICpHZXQqIEFQSSB0
byByZXRyaXZlIGRhdGENCj4gPj4+Pg0KPiA+Pj4+IFtyb290QGh1bnRlci1ub2RlMiByb2Nrc2Ri
XSMgLi9kYl9iZW5jaCAtdXNlX2V4aXN0aW5nX2RiDQo+ID4+Pj4gLWJlbmNobWFya3MgcmVhZHJh
bmRvbWZhc3QNCj4gPj4+Pg0KPiA+Pj4+IExldmVsREI6ICAgIHZlcnNpb24gNC4zDQo+ID4+Pj4g
RGF0ZTogICAgICAgV2VkIEp1biAgMSAwMDoyOToxNiAyMDE2DQo+ID4+Pj4gQ1BVOiAgICAgICAg
MzIgKiBJbnRlbChSKSBYZW9uKFIpIENQVSBFNS0yNjgwIDAgQCAyLjcwR0h6DQo+ID4+Pj4gQ1BV
Q2FjaGU6ICAgMjA0ODAgS0INCj4gPj4+PiBLZXlzOiAgICAgICAxNiBieXRlcyBlYWNoDQo+ID4+
Pj4gVmFsdWVzOiAgICAgMTAwIGJ5dGVzIGVhY2ggKDUwIGJ5dGVzIGFmdGVyIGNvbXByZXNzaW9u
KQ0KPiA+Pj4+IEVudHJpZXM6ICAgIDEwMDAwMDANCj4gPj4+PiBQcmVmaXg6ICAgIDAgYnl0ZXMN
Cj4gPj4+PiBLZXlzIHBlciBwcmVmaXg6ICAgIDANCj4gPj4+PiBSYXdTaXplOiAgICAxMTAuNiBN
QiAoZXN0aW1hdGVkKQ0KPiA+Pj4+IEZpbGVTaXplOiAgIDYyLjkgTUIgKGVzdGltYXRlZCkNCj4g
Pj4+PiBXcml0ZXMgcGVyIHNlY29uZDogMA0KPiA+Pj4+IENvbXByZXNzaW9uOiBTbmFwcHkNCj4g
Pj4+PiBNZW10YWJsZXJlcDogc2tpcF9saXN0DQo+ID4+Pj4gUGVyZiBMZXZlbDogMA0KPiA+Pj4+
IFdBUk5JTkc6IEFzc2VydGlvbnMgYXJlIGVuYWJsZWQ7IGJlbmNobWFya3MgdW5uZWNlc3Nhcmls
eSBzbG93DQo+ID4+Pj4gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tDQo+ID4+Pj4gREIgcGF0aDogWy90bXAvcm9ja3NkYnRlc3QtMC9kYmJlbmNoXQ0KPiA+
Pj4+IHJlYWRyYW5kb21mYXN0IDogICAgICAgNC41NzAgbWljcm9zL29wIDIxODgwNiBvcHMvc2Vj
OyAoMTAwMDEwMCBvZg0KPiA+Pj4+IDEwMDAxMDAgZm91bmQsIGlzc3VlZCA0NjYzOSBub24tZXhp
c3Qga2V5cykNCj4gPj4+Pg0KPiA+Pj4+ID09PT09PT09PT09PT09PT09PT09PT09PT09PQ0KPiA+
Pj4+IHRoZW4gSSBtb2RpZnkgcmVhZHJhbmRvbWZhc3QgdG8gdXNlIEl0ZXJhdG9yIEFQSVswXToN
Cj4gPj4+Pg0KPiA+Pj4+IFtyb290QGh1bnRlci1ub2RlMiByb2Nrc2RiXSMgLi9kYl9iZW5jaCAt
dXNlX2V4aXN0aW5nX2RiDQo+ID4+Pj4gLWJlbmNobWFya3MgcmVhZHJhbmRvbWZhc3QNCj4gPj4+
PiBMZXZlbERCOiAgICB2ZXJzaW9uIDQuMw0KPiA+Pj4+IERhdGU6ICAgICAgIFdlZCBKdW4gIDEg
MDA6MzM6MDMgMjAxNg0KPiA+Pj4+IENQVTogICAgICAgIDMyICogSW50ZWwoUikgWGVvbihSKSBD
UFUgRTUtMjY4MCAwIEAgMi43MEdIeg0KPiA+Pj4+IENQVUNhY2hlOiAgIDIwNDgwIEtCDQo+ID4+
Pj4gS2V5czogICAgICAgMTYgYnl0ZXMgZWFjaA0KPiA+Pj4+IFZhbHVlczogICAgIDEwMCBieXRl
cyBlYWNoICg1MCBieXRlcyBhZnRlciBjb21wcmVzc2lvbikNCj4gPj4+PiBFbnRyaWVzOiAgICAx
MDAwMDAwDQo+ID4+Pj4gUHJlZml4OiAgICAwIGJ5dGVzDQo+ID4+Pj4gS2V5cyBwZXIgcHJlZml4
OiAgICAwDQo+ID4+Pj4gUmF3U2l6ZTogICAgMTEwLjYgTUIgKGVzdGltYXRlZCkNCj4gPj4+PiBG
aWxlU2l6ZTogICA2Mi45IE1CIChlc3RpbWF0ZWQpDQo+ID4+Pj4gV3JpdGVzIHBlciBzZWNvbmQ6
IDANCj4gPj4+PiBDb21wcmVzc2lvbjogU25hcHB5DQo+ID4+Pj4gTWVtdGFibGVyZXA6IHNraXBf
bGlzdA0KPiA+Pj4+IFBlcmYgTGV2ZWw6IDANCj4gPj4+PiBXQVJOSU5HOiBBc3NlcnRpb25zIGFy
ZSBlbmFibGVkOyBiZW5jaG1hcmtzIHVubmVjZXNzYXJpbHkgc2xvdw0KPiA+Pj4+IC0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPiA+Pj4+IERCIHBhdGg6
IFsvdG1wL3JvY2tzZGJ0ZXN0LTAvZGJiZW5jaF0NCj4gPj4+PiByZWFkcmFuZG9tZmFzdCA6ICAg
ICAgNDUuMTg4IG1pY3Jvcy9vcCAyMjEyOSBvcHMvc2VjOyAoMTAwMDEwMCBvZg0KPiA+Pj4+IDEw
MDAxMDAgZm91bmQsIGlzc3VlZCA0NjYzOSBub24tZXhpc3Qga2V5cykNCj4gPj4+Pg0KPiA+Pj4+
DQo+ID4+Pj4gNDUuMTggdXMvb3AgdnMgNC41N3VzL29wIQ0KPiA+Pj4+DQo+ID4+Pj4gVGhlIHRl
c3QgY2FuIGJlIHJlcGVhdGVkIGFuZCBlYXN5IHRvIGRvISBQbHogY29ycmVjdCBpZiBJJ20gZG9p
bmcNCj4gPj4+PiBmb29saXNoIHRoaW5nIEknbSBub3QgYXdhcmUuLg0KPiA+Pj4+DQo+ID4+Pj4g
U28gSSBwcm9wb3NhbCB0aGlzIFBSOiBodHRwczovL2dpdGh1Yi5jb20vY2VwaC9jZXBoL3B1bGwv
OTQxMQ0KPiA+Pj4+DQo+ID4+Pj4gV2Ugc3RpbGwgY2FuIG1ha2UgZnVydGhlciBpbXByb3ZlbWVu
dHMgYnkgc2Nhbm5pbmcgYWxsIGl0ZXJhdGUNCj4gPj4+PiB1c2FnZSB0byBtYWtlIGl0IGJldHRl
ciENCj4gPj4+Pg0KPiA+Pj4+IFswXToNCj4gPj4+PiAtLS0gYS9kYi9kYl9iZW5jaC5jYw0KPiA+
Pj4+ICsrKyBiL2RiL2RiX2JlbmNoLmNjDQo+ID4+Pj4gQEAgLTI5MjMsMTQgKzI5MjMsMTIgQEAg
Y2xhc3MgQmVuY2htYXJrIHsNCj4gPj4+PiAgICAgICAgICBpbnQ2NF90IGtleV9yYW5kID0gdGhy
ZWFkLT5yYW5kLk5leHQoKSAmIChwb3QgLSAxKTsNCj4gPj4+PiAgICAgICAgICBHZW5lcmF0ZUtl
eUZyb21JbnQoa2V5X3JhbmQsIEZMQUdTX251bSwgJmtleSk7DQo+ID4+Pj4gICAgICAgICAgKyty
ZWFkOw0KPiA+Pj4+IC0gICAgICAgIGF1dG8gc3RhdHVzID0gZGItPkdldChvcHRpb25zLCBrZXks
ICZ2YWx1ZSk7DQo+ID4+Pj4gLSAgICAgICAgaWYgKHN0YXR1cy5vaygpKSB7DQo+ID4+Pj4gLSAg
ICAgICAgICArK2ZvdW5kOw0KPiA+Pj4+IC0gICAgICAgIH0gZWxzZSBpZiAoIXN0YXR1cy5Jc05v
dEZvdW5kKCkpIHsNCj4gPj4+PiAtICAgICAgICAgIGZwcmludGYoc3RkZXJyLCAiR2V0IHJldHVy
bmVkIGFuIGVycm9yOiAlc1xuIiwNCj4gPj4+PiAtICAgICAgICAgICAgICAgICAgc3RhdHVzLlRv
U3RyaW5nKCkuY19zdHIoKSk7DQo+ID4+Pj4gLSAgICAgICAgICBhYm9ydCgpOw0KPiA+Pj4+IC0g
ICAgICAgIH0NCj4gPj4+PiArICAgICAgICBJdGVyYXRvciogaXRlciA9IGRiLT5OZXdJdGVyYXRv
cihvcHRpb25zKTsNCj4gPj4+PiArICAgICAgaXRlci0+U2VlayhrZXkpOw0KPiA+Pj4+ICsgICAg
ICBpZiAoaXRlci0+VmFsaWQoKSAmJiBpdGVyLT5rZXkoKS5jb21wYXJlKGtleSkgPT0gMCkgew0K
PiA+Pj4+ICsgICAgICAgIGZvdW5kKys7DQo+ID4+Pj4gKyAgICAgIH0NCj4gPj4+PiArDQo+ID4+
Pj4gICAgICAgICAgaWYgKGtleV9yYW5kID49IEZMQUdTX251bSkgew0KPiA+Pj4+ICAgICAgICAg
ICAgKytub25leGlzdDsNCj4gPj4+PiAgICAgICAgICB9DQo+ID4+Pj4gLS0NCj4gPj4+PiBUbyB1
bnN1YnNjcmliZSBmcm9tIHRoaXMgbGlzdDogc2VuZCB0aGUgbGluZSAidW5zdWJzY3JpYmUNCj4g
Pj4+PiBjZXBoLWRldmVsIiBpbiB0aGUgYm9keSBvZiBhIG1lc3NhZ2UgdG8gbWFqb3Jkb21vQHZn
ZXIua2VybmVsLm9yZw0KPiA+Pj4+IE1vcmUgbWFqb3Jkb21vIGluZm8gYXQgIGh0dHA6Ly92Z2Vy
Lmtlcm5lbC5vcmcvbWFqb3Jkb21vLWluZm8uaHRtbA0KPiA+PiAtLQ0KPiA+PiBUbyB1bnN1YnNj
cmliZSBmcm9tIHRoaXMgbGlzdDogc2VuZCB0aGUgbGluZSAidW5zdWJzY3JpYmUgY2VwaC1kZXZl
bCINCj4gPj4gaW4gdGhlIGJvZHkgb2YgYSBtZXNzYWdlIHRvIG1ham9yZG9tb0B2Z2VyLmtlcm5l
bC5vcmcgTW9yZQ0KPiBtYWpvcmRvbW8NCj4gPj4gaW5mbyBhdCAgaHR0cDovL3ZnZXIua2VybmVs
Lm9yZy9tYWpvcmRvbW8taW5mby5odG1sDQo+IC0tDQo+IFRvIHVuc3Vic2NyaWJlIGZyb20gdGhp
cyBsaXN0OiBzZW5kIHRoZSBsaW5lICJ1bnN1YnNjcmliZSBjZXBoLWRldmVsIiBpbiB0aGUNCj4g
Ym9keSBvZiBhIG1lc3NhZ2UgdG8gbWFqb3Jkb21vQHZnZXIua2VybmVsLm9yZyBNb3JlIG1ham9y
ZG9tbyBpbmZvIGF0DQo+IGh0dHA6Ly92Z2VyLmtlcm5lbC5vcmcvbWFqb3Jkb21vLWluZm8uaHRt
bA0K
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Jianjian Huo June 1, 2016, 3:13 a.m. UTC | #9
DQpPbiBUdWUsIE1heSAzMSwgMjAxNiBhdCA3OjM4IFBNLCBIYW9tYWkgV2FuZyA8aGFvbWFpQHhz
a3kuY29tPiB3cm90ZToNCj4gT24gV2VkLCBKdW4gMSwgMjAxNiBhdCAzOjQwIEFNLCBKaWFuamlh
biBIdW8gPGppYW5qaWFuLmh1b0BzYW1zdW5nLmNvbT4gd3JvdGU6DQo+Pg0KPj4gT24gVHVlLCBN
YXkgMzEsIDIwMTYgYXQgMTE6MTIgQU0sIEhhb21haSBXYW5nIDxoYW9tYWlAeHNreS5jb20+IHdy
b3RlOg0KPj4+IE9uIFdlZCwgSnVuIDEsIDIwMTYgYXQgMjowOCBBTSwgSmlhbmppYW4gSHVvIDxq
aWFuamlhbi5odW9Ac2Ftc3VuZy5jb20+IHdyb3RlOg0KPj4+PiBIaSBIYW9tYWksDQo+Pj4+DQo+
Pj4+IEkgbm90aWNlZCB0aGlzIGFzIHdlbGwsIGFuZCBtYWRlIHNhbWUgY2hhbmdlcyB0byBSb2Nr
c0RCU3RvcmUgaW4gdGhpcyBQUiBsYXN0IHdlZWs6DQo+Pj4+IGh0dHBzOi8vZ2l0aHViLmNvbS9j
ZXBoL2NlcGgvcHVsbC85MjE1DQo+Pj4+DQo+Pj4+IE9uZSB0aGluZyB3aGljaCBpcyBldmVuIHdv
cnNlLCAgc2VlayB3aWxsIGJ5cGFzcyByb3cgY2FjaGUsIHNvIGt2IHBhaXJzIHdvbid0IGJlIGFi
bGUgdG8gYmUgY2FjaGVkIGluIHJvdyBjYWNoZS4NCj4+Pj4gSSBhbSB3b3JraW5nIHRvIGJlbmNo
bWFyayB0aGUgcGVyZm9ybWFuY2UgaW1wYWN0LCB3aWxsIHB1Ymxpc2ggdGhlIHJlc3VsdHMgYWZ0
ZXIgSSBhbSBkb25lIHRoaXMgd2Vlay4NCj4+Pg0KPj4+IE9oLCBjb29sISBJIHRoaW5rIHlvdSBj
YW4gY2hlcnJ5LXBpY2sgbXkgYW5vdGhlciBsZXZlbGRiIGZpeC4NCj4+DQo+PiBObyBwcm9ibGVt
LCBJIHdpbGwgZG8gdGhhdC4NCj4+Pg0KPj4+IEJUVywgZG8geW91IHBheSBhbiBhdHRlbnRpb24g
dG8gcHJlZml4IHNlZWsgYXBpPyBJIHRoaW5rIGl0IHdpbGwgYmUNCj4+PiBtb3JlIHN1aXRhYmxl
IHRoYW4gY29sdW1uIGZhbWlseSBpbiBjZXBoIGNhc2UuIElmIHdlIGNhbiBoYXZlDQo+Pj4gd2Vs
bC1kZWZpbmVkIHByZWZpeCBydWxlLCB3ZSBjYW4gbWFrZSBtb3N0IG9mIHJhbmdlIHF1ZXJ5IGNo
ZWFwZXIhDQo+Pg0KPj4gV2hlbiBrZXlzIHdpdGggc2FtZSBwcmVmaXggYXJlIHN0b3JlZCBpbiB0
aGVpciBvd24gU1NUIGZpbGVzKENGIGNhc2UpLCBldmVuIHNlZWtpbmcgd2l0aG91dCBwcmVmaXgg
d2lsbCBiZSBmYXN0ZXIgdGhhbiBzZWVraW5nIHdpdGggcHJlZml4IGJ1dCBtaXhlZCB3aXRoIGRp
ZmZlcmVudCBwcmVmaXhlZCBrZXlzPw0KPg0KPiBGcm9tIG15IGN1cnJlbnQgdmlldywgcHJlZml4
IGNvdWxkIGJlIG1vcmUgZmxleGlibGUuIEZvciBleGFtcGxlLCBlYWNoDQo+IHJndyBidWNrZXQg
aW5kZXggY291bGQgdXNlIG9uZSBwcmVmaXggdG8gbWFrZSBlYWNoIGJ1Y2tldCBpbmRleCBvYmpl
Y3QNCj4gc2VlayBzZXBhcmF0ZWQuIEZvciBDRiwgaXQgd291bGQgYmUgdG9vIGhlYXZyeS4NCg0K
U3VyZSwgbm8gZG91YnQgcHJlZml4IHNlZWsgd2lsbCBoZWxwIHNvbWUgY2FzZXMsIENGIHdpbGwg
aGVscCBvdGhlcnMuIEFuZCBwcmVmaXggd2lsbCBtb3N0bHkgYmVuZWZpdCBzZWVrLCB3aGlsZSBD
RiBoYXMgYnJvYWRlciBpbXBhY3QuDQo+DQo+PiBJIGFtIG5vdCBzdXJlIHdoYXQgb3B0aW1pemF0
aW9uIHByZWZpeCBzZWVrIHdpbGwgdXNlIGludGVybmFsbHkgZm9yIGJsb2NrIGJhc2VkIGZvcm1h
dCwgYnV0IHRvIG1lLCBpdCdzIGhhcmQgdG8gYmVhdCB0aGUgY2FzZSB3aGVuIHlvdSBvbmx5IGhh
dmUgb25lIHByZWZpeGVkIGtleXMgc3RvcmVkIHNlcGFyYXRlbHkuDQo+Pj4NCj4+Pj4NCj4+Pj4g
SmlhbmppYW4NCj4+Pj4NCj4+Pj4gT24gVHVlLCBNYXkgMzEsIDIwMTYgYXQgOTo0OSBBTSwgSGFv
bWFpIFdhbmcgPGhhb21haUB4c2t5LmNvbT4gd3JvdGU6DQo+Pj4+PiBIaSBTYWdlIGFuZCBNYXJr
LA0KPj4+Pj4NCj4+Pj4+IEFzIG1lbnRpb25lZCBpbiBCbHVlU3RvcmUgc3RhbmR1cCwgSSBmb3Vu
ZCByb2Nrc2RiIGl0ZXJhdG9yICpTZWVrKg0KPj4+Pj4gd29uJ3QgdXNlIGJsb29tIGZpbHRlciBs
aWtlICpHZXQqLg0KPj4+Pj4NCj4+Pj4+ICpHZXQqIGltcGw6IGl0IHdpbGwgbG9vayBhdCBmaWx0
ZXIgZmlyc3RseQ0KPj4+Pj4gaHR0cHM6Ly9naXRodWIuY29tL2ZhY2Vib29rL3JvY2tzZGIvYmxv
Yi9tYXN0ZXIvdGFibGUvYmxvY2tfYmFzZWRfdGFibGVfcmVhZGVyLmNjI0wxMzY5DQo+Pj4+Pg0K
Pj4+Pj4gSXRlcmF0b3IgKlNlZWsqOiBpdCB3aWxsIGRvIGJpbmFyeSBzZWFyY2gsIGJ5IGRlZmF1
bHQgd2UgZG9uJ3Qgc3BlY2lmeQ0KPj4+Pj4gcHJlZml4IGZlYXR1cmUoaHR0cHM6Ly9naXRodWIu
Y29tL2ZhY2Vib29rL3JvY2tzZGIvd2lraS9QcmVmaXgtU2Vlay1BUEktQ2hhbmdlcykuDQo+Pj4+
PiBodHRwczovL2dpdGh1Yi5jb20vZmFjZWJvb2svcm9ja3NkYi9ibG9iL21hc3Rlci90YWJsZS9i
bG9jay5jYyNMOTQNCj4+Pj4+DQo+Pj4+PiBTbyBJIHVzZSBhIHNpbXBsZSB0ZXN0czoNCj4+Pj4+
DQo+Pj4+PiAuL2RiX2JlbmNoIC1udW0gMTAwMDAwMDAgIC1iZW5jaG1hcmtzIGZpbGxiYXRjaA0K
Pj4+Pj4gZmlsbCB0aGUgZGIgZmlyc3RseSB3aXRoIDEwMDB3IHJlY29yZHMuDQo+Pj4+Pg0KPj4+
Pj4gLi9kYl9iZW5jaCAtdXNlX2V4aXN0aW5nX2RiICAtYmVuY2htYXJrcyByZWFkcmFuZG9tZmFz
dA0KPj4+Pj4gcmVhZHJhbmRvbWZhc3QgY2FzZSB3aWxsIHVzZSAqR2V0KiBBUEkgdG8gcmV0cml2
ZSBkYXRhDQo+Pj4+Pg0KPj4+Pj4gW3Jvb3RAaHVudGVyLW5vZGUyIHJvY2tzZGJdIyAuL2RiX2Jl
bmNoIC11c2VfZXhpc3RpbmdfZGIgIC1iZW5jaG1hcmtzDQo+Pj4+PiByZWFkcmFuZG9tZmFzdA0K
Pj4+Pj4NCj4+Pj4+IExldmVsREI6ICAgIHZlcnNpb24gNC4zDQo+Pj4+PiBEYXRlOiAgICAgICBX
ZWQgSnVuICAxIDAwOjI5OjE2IDIwMTYNCj4+Pj4+IENQVTogICAgICAgIDMyICogSW50ZWwoUikg
WGVvbihSKSBDUFUgRTUtMjY4MCAwIEAgMi43MEdIeg0KPj4+Pj4gQ1BVQ2FjaGU6ICAgMjA0ODAg
S0INCj4+Pj4+IEtleXM6ICAgICAgIDE2IGJ5dGVzIGVhY2gNCj4+Pj4+IFZhbHVlczogICAgIDEw
MCBieXRlcyBlYWNoICg1MCBieXRlcyBhZnRlciBjb21wcmVzc2lvbikNCj4+Pj4+IEVudHJpZXM6
ICAgIDEwMDAwMDANCj4+Pj4+IFByZWZpeDogICAgMCBieXRlcw0KPj4+Pj4gS2V5cyBwZXIgcHJl
Zml4OiAgICAwDQo+Pj4+PiBSYXdTaXplOiAgICAxMTAuNiBNQiAoZXN0aW1hdGVkKQ0KPj4+Pj4g
RmlsZVNpemU6ICAgNjIuOSBNQiAoZXN0aW1hdGVkKQ0KPj4+Pj4gV3JpdGVzIHBlciBzZWNvbmQ6
IDANCj4+Pj4+IENvbXByZXNzaW9uOiBTbmFwcHkNCj4+Pj4+IE1lbXRhYmxlcmVwOiBza2lwX2xp
c3QNCj4+Pj4+IFBlcmYgTGV2ZWw6IDANCj4+Pj4+IFdBUk5JTkc6IEFzc2VydGlvbnMgYXJlIGVu
YWJsZWQ7IGJlbmNobWFya3MgdW5uZWNlc3NhcmlseSBzbG93DQo+Pj4+PiAtLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCj4+Pj4+IERCIHBhdGg6IFsvdG1w
L3JvY2tzZGJ0ZXN0LTAvZGJiZW5jaF0NCj4+Pj4+IHJlYWRyYW5kb21mYXN0IDogICAgICAgNC41
NzAgbWljcm9zL29wIDIxODgwNiBvcHMvc2VjOyAoMTAwMDEwMCBvZg0KPj4+Pj4gMTAwMDEwMCBm
b3VuZCwgaXNzdWVkIDQ2NjM5IG5vbi1leGlzdCBrZXlzKQ0KPj4+Pj4NCj4+Pj4+ID09PT09PT09
PT09PT09PT09PT09PT09PT09PQ0KPj4+Pj4gdGhlbiBJIG1vZGlmeSByZWFkcmFuZG9tZmFzdCB0
byB1c2UgSXRlcmF0b3IgQVBJWzBdOg0KPj4+Pj4NCj4+Pj4+IFtyb290QGh1bnRlci1ub2RlMiBy
b2Nrc2RiXSMgLi9kYl9iZW5jaCAtdXNlX2V4aXN0aW5nX2RiICAtYmVuY2htYXJrcw0KPj4+Pj4g
cmVhZHJhbmRvbWZhc3QNCj4+Pj4+IExldmVsREI6ICAgIHZlcnNpb24gNC4zDQo+Pj4+PiBEYXRl
OiAgICAgICBXZWQgSnVuICAxIDAwOjMzOjAzIDIwMTYNCj4+Pj4+IENQVTogICAgICAgIDMyICog
SW50ZWwoUikgWGVvbihSKSBDUFUgRTUtMjY4MCAwIEAgMi43MEdIeg0KPj4+Pj4gQ1BVQ2FjaGU6
ICAgMjA0ODAgS0INCj4+Pj4+IEtleXM6ICAgICAgIDE2IGJ5dGVzIGVhY2gNCj4+Pj4+IFZhbHVl
czogICAgIDEwMCBieXRlcyBlYWNoICg1MCBieXRlcyBhZnRlciBjb21wcmVzc2lvbikNCj4+Pj4+
IEVudHJpZXM6ICAgIDEwMDAwMDANCj4+Pj4+IFByZWZpeDogICAgMCBieXRlcw0KPj4+Pj4gS2V5
cyBwZXIgcHJlZml4OiAgICAwDQo+Pj4+PiBSYXdTaXplOiAgICAxMTAuNiBNQiAoZXN0aW1hdGVk
KQ0KPj4+Pj4gRmlsZVNpemU6ICAgNjIuOSBNQiAoZXN0aW1hdGVkKQ0KPj4+Pj4gV3JpdGVzIHBl
ciBzZWNvbmQ6IDANCj4+Pj4+IENvbXByZXNzaW9uOiBTbmFwcHkNCj4+Pj4+IE1lbXRhYmxlcmVw
OiBza2lwX2xpc3QNCj4+Pj4+IFBlcmYgTGV2ZWw6IDANCj4+Pj4+IFdBUk5JTkc6IEFzc2VydGlv
bnMgYXJlIGVuYWJsZWQ7IGJlbmNobWFya3MgdW5uZWNlc3NhcmlseSBzbG93DQo+Pj4+PiAtLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCj4+Pj4+IERCIHBh
dGg6IFsvdG1wL3JvY2tzZGJ0ZXN0LTAvZGJiZW5jaF0NCj4+Pj4+IHJlYWRyYW5kb21mYXN0IDog
ICAgICA0NS4xODggbWljcm9zL29wIDIyMTI5IG9wcy9zZWM7ICgxMDAwMTAwIG9mDQo+Pj4+PiAx
MDAwMTAwIGZvdW5kLCBpc3N1ZWQgNDY2Mzkgbm9uLWV4aXN0IGtleXMpDQo+Pj4+Pg0KPj4+Pj4N
Cj4+Pj4+IDQ1LjE4IHVzL29wIHZzIDQuNTd1cy9vcCENCj4+Pj4+DQo+Pj4+PiBUaGUgdGVzdCBj
YW4gYmUgcmVwZWF0ZWQgYW5kIGVhc3kgdG8gZG8hIFBseiBjb3JyZWN0IGlmIEknbSBkb2luZw0K
Pj4+Pj4gZm9vbGlzaCB0aGluZyBJJ20gbm90IGF3YXJlLi4NCj4+Pj4+DQo+Pj4+PiBTbyBJIHBy
b3Bvc2FsIHRoaXMgUFI6IGh0dHBzOi8vZ2l0aHViLmNvbS9jZXBoL2NlcGgvcHVsbC85NDExDQo+
Pj4+Pg0KPj4+Pj4gV2Ugc3RpbGwgY2FuIG1ha2UgZnVydGhlciBpbXByb3ZlbWVudHMgYnkgc2Nh
bm5pbmcgYWxsIGl0ZXJhdGUgdXNhZ2UNCj4+Pj4+IHRvIG1ha2UgaXQgYmV0dGVyIQ0KPj4+Pj4N
Cj4+Pj4+IFswXToNCj4+Pj4+IC0tLSBhL2RiL2RiX2JlbmNoLmNjDQo+Pj4+PiArKysgYi9kYi9k
Yl9iZW5jaC5jYw0KPj4+Pj4gQEAgLTI5MjMsMTQgKzI5MjMsMTIgQEAgY2xhc3MgQmVuY2htYXJr
IHsNCj4+Pj4+ICAgICAgICAgIGludDY0X3Qga2V5X3JhbmQgPSB0aHJlYWQtPnJhbmQuTmV4dCgp
ICYgKHBvdCAtIDEpOw0KPj4+Pj4gICAgICAgICAgR2VuZXJhdGVLZXlGcm9tSW50KGtleV9yYW5k
LCBGTEFHU19udW0sICZrZXkpOw0KPj4+Pj4gICAgICAgICAgKytyZWFkOw0KPj4+Pj4gLSAgICAg
ICAgYXV0byBzdGF0dXMgPSBkYi0+R2V0KG9wdGlvbnMsIGtleSwgJnZhbHVlKTsNCj4+Pj4+IC0g
ICAgICAgIGlmIChzdGF0dXMub2soKSkgew0KPj4+Pj4gLSAgICAgICAgICArK2ZvdW5kOw0KPj4+
Pj4gLSAgICAgICAgfSBlbHNlIGlmICghc3RhdHVzLklzTm90Rm91bmQoKSkgew0KPj4+Pj4gLSAg
ICAgICAgICBmcHJpbnRmKHN0ZGVyciwgIkdldCByZXR1cm5lZCBhbiBlcnJvcjogJXNcbiIsDQo+
Pj4+PiAtICAgICAgICAgICAgICAgICAgc3RhdHVzLlRvU3RyaW5nKCkuY19zdHIoKSk7DQo+Pj4+
PiAtICAgICAgICAgIGFib3J0KCk7DQo+Pj4+PiAtICAgICAgICB9DQo+Pj4+PiArICAgICAgICBJ
dGVyYXRvciogaXRlciA9IGRiLT5OZXdJdGVyYXRvcihvcHRpb25zKTsNCj4+Pj4+ICsgICAgICBp
dGVyLT5TZWVrKGtleSk7DQo+Pj4+PiArICAgICAgaWYgKGl0ZXItPlZhbGlkKCkgJiYgaXRlci0+
a2V5KCkuY29tcGFyZShrZXkpID09IDApIHsNCj4+Pj4+ICsgICAgICAgIGZvdW5kKys7DQo+Pj4+
PiArICAgICAgfQ0KPj4+Pj4gKw0KPj4+Pj4gICAgICAgICAgaWYgKGtleV9yYW5kID49IEZMQUdT
X251bSkgew0KPj4+Pj4gICAgICAgICAgICArK25vbmV4aXN0Ow0KPj4+Pj4gICAgICAgICAgfQ0K
Pj4+Pj4gLS0NCj4+Pj4+IFRvIHVuc3Vic2NyaWJlIGZyb20gdGhpcyBsaXN0OiBzZW5kIHRoZSBs
aW5lICJ1bnN1YnNjcmliZSBjZXBoLWRldmVsIiBpbg0KPj4+Pj4gdGhlIGJvZHkgb2YgYSBtZXNz
YWdlIHRvIG1ham9yZG9tb0B2Z2VyLmtlcm5lbC5vcmcNCj4+Pj4+IE1vcmUgbWFqb3Jkb21vIGlu
Zm8gYXQgIGh0dHA6Ly92Z2VyLmtlcm5lbC5vcmcvbWFqb3Jkb21vLWluZm8uaHRtbA0KPj4+IC0t
DQo+Pj4gVG8gdW5zdWJzY3JpYmUgZnJvbSB0aGlzIGxpc3Q6IHNlbmQgdGhlIGxpbmUgInVuc3Vi
c2NyaWJlIGNlcGgtZGV2ZWwiIGluDQo+Pj4gdGhlIGJvZHkgb2YgYSBtZXNzYWdlIHRvIG1ham9y
ZG9tb0B2Z2VyLmtlcm5lbC5vcmcNCj4+PiBNb3JlIG1ham9yZG9tbyBpbmZvIGF0ICBodHRwOi8v
dmdlci5rZXJuZWwub3JnL21ham9yZG9tby1pbmZvLmh0bWwNCj4gLS0NCj4gVG8gdW5zdWJzY3Jp
YmUgZnJvbSB0aGlzIGxpc3Q6IHNlbmQgdGhlIGxpbmUgInVuc3Vic2NyaWJlIGNlcGgtZGV2ZWwi
IGluDQo+IHRoZSBib2R5IG9mIGEgbWVzc2FnZSB0byBtYWpvcmRvbW9Admdlci5rZXJuZWwub3Jn
DQo+IE1vcmUgbWFqb3Jkb21vIGluZm8gYXQgIGh0dHA6Ly92Z2VyLmtlcm5lbC5vcmcvbWFqb3Jk
b21vLWluZm8uaHRtbA0K
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Jianjian Huo June 1, 2016, 3:24 a.m. UTC | #10
SGkgQWxsZW4sDQoNCk9uIFR1ZSwgTWF5IDMxLCAyMDE2IGF0IDc6NTIgUE0sIEFsbGVuIFNhbXVl
bHMgPEFsbGVuLlNhbXVlbHNAc2FuZGlzay5jb20+IHdyb3RlOg0KPj4gLS0tLS1PcmlnaW5hbCBN
ZXNzYWdlLS0tLS0NCj4+IEZyb206IGNlcGgtZGV2ZWwtb3duZXJAdmdlci5rZXJuZWwub3JnIFtt
YWlsdG86Y2VwaC1kZXZlbC0NCj4+IG93bmVyQHZnZXIua2VybmVsLm9yZ10gT24gQmVoYWxmIE9m
IEhhb21haSBXYW5nDQo+PiBTZW50OiBUdWVzZGF5LCBNYXkgMzEsIDIwMTYgNzozOCBQTQ0KPj4g
VG86IEppYW5qaWFuIEh1byA8amlhbmppYW4uaHVvQHNhbXN1bmcuY29tPg0KPj4gQ2M6IFNhZ2Ug
V2VpbCA8c3dlaWxAcmVkaGF0LmNvbT47IE1hcmsgTmVsc29uIDxtbmVsc29uQHJlZGhhdC5jb20+
Ow0KPj4gY2VwaC1kZXZlbEB2Z2VyLmtlcm5lbC5vcmcNCj4+IFN1YmplY3Q6IFJlOiBSb2Nrc0RC
IEluY29ycmVjdCBBUEkgVXNhZ2UNCj4+DQo+PiBPbiBXZWQsIEp1biAxLCAyMDE2IGF0IDM6NDAg
QU0sIEppYW5qaWFuIEh1byA8amlhbmppYW4uaHVvQHNhbXN1bmcuY29tPg0KPj4gd3JvdGU6DQo+
PiA+DQo+PiA+IE9uIFR1ZSwgTWF5IDMxLCAyMDE2IGF0IDExOjEyIEFNLCBIYW9tYWkgV2FuZyA8
aGFvbWFpQHhza3kuY29tPg0KPj4gd3JvdGU6DQo+PiA+PiBPbiBXZWQsIEp1biAxLCAyMDE2IGF0
IDI6MDggQU0sIEppYW5qaWFuIEh1bw0KPj4gPGppYW5qaWFuLmh1b0BzYW1zdW5nLmNvbT4gd3Jv
dGU6DQo+PiA+Pj4gSGkgSGFvbWFpLA0KPj4gPj4+DQo+PiA+Pj4gSSBub3RpY2VkIHRoaXMgYXMg
d2VsbCwgYW5kIG1hZGUgc2FtZSBjaGFuZ2VzIHRvIFJvY2tzREJTdG9yZSBpbiB0aGlzIFBSDQo+
PiBsYXN0IHdlZWs6DQo+PiA+Pj4gaHR0cHM6Ly9naXRodWIuY29tL2NlcGgvY2VwaC9wdWxsLzky
MTUNCj4+ID4+Pg0KPj4gPj4+IE9uZSB0aGluZyB3aGljaCBpcyBldmVuIHdvcnNlLCAgc2VlayB3
aWxsIGJ5cGFzcyByb3cgY2FjaGUsIHNvIGt2IHBhaXJzDQo+PiB3b24ndCBiZSBhYmxlIHRvIGJl
IGNhY2hlZCBpbiByb3cgY2FjaGUuDQo+PiA+Pj4gSSBhbSB3b3JraW5nIHRvIGJlbmNobWFyayB0
aGUgcGVyZm9ybWFuY2UgaW1wYWN0LCB3aWxsIHB1Ymxpc2ggdGhlDQo+PiByZXN1bHRzIGFmdGVy
IEkgYW0gZG9uZSB0aGlzIHdlZWsuDQo+PiA+Pg0KPj4gPj4gT2gsIGNvb2whIEkgdGhpbmsgeW91
IGNhbiBjaGVycnktcGljayBteSBhbm90aGVyIGxldmVsZGIgZml4Lg0KPj4gPg0KPj4gPiBObyBw
cm9ibGVtLCBJIHdpbGwgZG8gdGhhdC4NCj4+ID4+DQo+PiA+PiBCVFcsIGRvIHlvdSBwYXkgYW4g
YXR0ZW50aW9uIHRvIHByZWZpeCBzZWVrIGFwaT8gSSB0aGluayBpdCB3aWxsIGJlDQo+PiA+PiBt
b3JlIHN1aXRhYmxlIHRoYW4gY29sdW1uIGZhbWlseSBpbiBjZXBoIGNhc2UuIElmIHdlIGNhbiBo
YXZlDQo+PiA+PiB3ZWxsLWRlZmluZWQgcHJlZml4IHJ1bGUsIHdlIGNhbiBtYWtlIG1vc3Qgb2Yg
cmFuZ2UgcXVlcnkgY2hlYXBlciENCj4+ID4NCj4+ID4gV2hlbiBrZXlzIHdpdGggc2FtZSBwcmVm
aXggYXJlIHN0b3JlZCBpbiB0aGVpciBvd24gU1NUIGZpbGVzKENGIGNhc2UpLA0KPj4gZXZlbiBz
ZWVraW5nIHdpdGhvdXQgcHJlZml4IHdpbGwgYmUgZmFzdGVyIHRoYW4gc2Vla2luZyB3aXRoIHBy
ZWZpeCBidXQgbWl4ZWQNCj4+IHdpdGggZGlmZmVyZW50IHByZWZpeGVkIGtleXM/DQo+Pg0KPj4g
RnJvbSBteSBjdXJyZW50IHZpZXcsIHByZWZpeCBjb3VsZCBiZSBtb3JlIGZsZXhpYmxlLiBGb3Ig
ZXhhbXBsZSwgZWFjaCByZ3cNCj4+IGJ1Y2tldCBpbmRleCBjb3VsZCB1c2Ugb25lIHByZWZpeCB0
byBtYWtlIGVhY2ggYnVja2V0IGluZGV4IG9iamVjdCBzZWVrDQo+PiBzZXBhcmF0ZWQuIEZvciBD
RiwgaXQgd291bGQgYmUgdG9vIGhlYXZyeS4NCj4NCj4gVGhpcyBtaWdodCBjYXVzZSBvdGhlciBw
cm9ibGVtcy4gV291bGQgaXQgZHJhbWF0aWNhbGx5IGluY3JlYXNlIHRoZSBudW1iZXIgb2YgZmls
ZXMgdGhhdCBCbHVlRlMgbmVlZHMgdG8gbWFuYWdlPyBJZiBzbywgdGhhdCBtaWdodCBlZmZlY3Rp
dmVseSBicmVhayB0aGF0IGNvZGUgdG9vIChvZiBjb3Vyc2UgaXQncyBmaXhhYmxlIGFsc28gOikp
DQoNClRoYW5rcyBmb3IgYnJpbmdpbmcgdXAgdGhpcyBpc3N1ZS4gV2hhdCdzIGN1cnJlbnQgbGlt
aXQgb2YgbnVtYmVyIG9mIGZpbGVzIGZvciBCbHVlRlMsIGZyb20geW91ciBleHBlcmllbmNlPyAN
CkppYW5qaWFuDQo+Pg0KPj4gPiBJIGFtIG5vdCBzdXJlIHdoYXQgb3B0aW1pemF0aW9uIHByZWZp
eCBzZWVrIHdpbGwgdXNlIGludGVybmFsbHkgZm9yIGJsb2NrDQo+PiBiYXNlZCBmb3JtYXQsIGJ1
dCB0byBtZSwgaXQncyBoYXJkIHRvIGJlYXQgdGhlIGNhc2Ugd2hlbiB5b3Ugb25seSBoYXZlIG9u
ZQ0KPj4gcHJlZml4ZWQga2V5cyBzdG9yZWQgc2VwYXJhdGVseS4NCj4+ID4+DQo+PiA+Pj4NCj4+
ID4+PiBKaWFuamlhbg0KPj4gPj4+DQo+PiA+Pj4gT24gVHVlLCBNYXkgMzEsIDIwMTYgYXQgOTo0
OSBBTSwgSGFvbWFpIFdhbmcgPGhhb21haUB4c2t5LmNvbT4NCj4+IHdyb3RlOg0KPj4gPj4+PiBI
aSBTYWdlIGFuZCBNYXJrLA0KPj4gPj4+Pg0KPj4gPj4+PiBBcyBtZW50aW9uZWQgaW4gQmx1ZVN0
b3JlIHN0YW5kdXAsIEkgZm91bmQgcm9ja3NkYiBpdGVyYXRvciAqU2VlayoNCj4+ID4+Pj4gd29u
J3QgdXNlIGJsb29tIGZpbHRlciBsaWtlICpHZXQqLg0KPj4gPj4+Pg0KPj4gPj4+PiAqR2V0KiBp
bXBsOiBpdCB3aWxsIGxvb2sgYXQgZmlsdGVyIGZpcnN0bHkNCj4+ID4+Pj4NCj4+IGh0dHBzOi8v
Z2l0aHViLmNvbS9mYWNlYm9vay9yb2Nrc2RiL2Jsb2IvbWFzdGVyL3RhYmxlL2Jsb2NrX2Jhc2Vk
X3QNCj4+ID4+Pj4gYWJsZV9yZWFkZXIuY2MjTDEzNjkNCj4+ID4+Pj4NCj4+ID4+Pj4gSXRlcmF0
b3IgKlNlZWsqOiBpdCB3aWxsIGRvIGJpbmFyeSBzZWFyY2gsIGJ5IGRlZmF1bHQgd2UgZG9uJ3QN
Cj4+ID4+Pj4gc3BlY2lmeSBwcmVmaXgNCj4+IGZlYXR1cmUoaHR0cHM6Ly9naXRodWIuY29tL2Zh
Y2Vib29rL3JvY2tzZGIvd2lraS9QcmVmaXgtU2Vlay1BUEktDQo+PiBDaGFuZ2VzKS4NCj4+ID4+
Pj4gaHR0cHM6Ly9naXRodWIuY29tL2ZhY2Vib29rL3JvY2tzZGIvYmxvYi9tYXN0ZXIvdGFibGUv
YmxvY2suY2MjTDk0DQo+PiA+Pj4+DQo+PiA+Pj4+IFNvIEkgdXNlIGEgc2ltcGxlIHRlc3RzOg0K
Pj4gPj4+Pg0KPj4gPj4+PiAuL2RiX2JlbmNoIC1udW0gMTAwMDAwMDAgIC1iZW5jaG1hcmtzIGZp
bGxiYXRjaCBmaWxsIHRoZSBkYiBmaXJzdGx5DQo+PiA+Pj4+IHdpdGggMTAwMHcgcmVjb3Jkcy4N
Cj4+ID4+Pj4NCj4+ID4+Pj4gLi9kYl9iZW5jaCAtdXNlX2V4aXN0aW5nX2RiICAtYmVuY2htYXJr
cyByZWFkcmFuZG9tZmFzdA0KPj4gPj4+PiByZWFkcmFuZG9tZmFzdCBjYXNlIHdpbGwgdXNlICpH
ZXQqIEFQSSB0byByZXRyaXZlIGRhdGENCj4+ID4+Pj4NCj4+ID4+Pj4gW3Jvb3RAaHVudGVyLW5v
ZGUyIHJvY2tzZGJdIyAuL2RiX2JlbmNoIC11c2VfZXhpc3RpbmdfZGINCj4+ID4+Pj4gLWJlbmNo
bWFya3MgcmVhZHJhbmRvbWZhc3QNCj4+ID4+Pj4NCj4+ID4+Pj4gTGV2ZWxEQjogICAgdmVyc2lv
biA0LjMNCj4+ID4+Pj4gRGF0ZTogICAgICAgV2VkIEp1biAgMSAwMDoyOToxNiAyMDE2DQo+PiA+
Pj4+IENQVTogICAgICAgIDMyICogSW50ZWwoUikgWGVvbihSKSBDUFUgRTUtMjY4MCAwIEAgMi43
MEdIeg0KPj4gPj4+PiBDUFVDYWNoZTogICAyMDQ4MCBLQg0KPj4gPj4+PiBLZXlzOiAgICAgICAx
NiBieXRlcyBlYWNoDQo+PiA+Pj4+IFZhbHVlczogICAgIDEwMCBieXRlcyBlYWNoICg1MCBieXRl
cyBhZnRlciBjb21wcmVzc2lvbikNCj4+ID4+Pj4gRW50cmllczogICAgMTAwMDAwMA0KPj4gPj4+
PiBQcmVmaXg6ICAgIDAgYnl0ZXMNCj4+ID4+Pj4gS2V5cyBwZXIgcHJlZml4OiAgICAwDQo+PiA+
Pj4+IFJhd1NpemU6ICAgIDExMC42IE1CIChlc3RpbWF0ZWQpDQo+PiA+Pj4+IEZpbGVTaXplOiAg
IDYyLjkgTUIgKGVzdGltYXRlZCkNCj4+ID4+Pj4gV3JpdGVzIHBlciBzZWNvbmQ6IDANCj4+ID4+
Pj4gQ29tcHJlc3Npb246IFNuYXBweQ0KPj4gPj4+PiBNZW10YWJsZXJlcDogc2tpcF9saXN0DQo+
PiA+Pj4+IFBlcmYgTGV2ZWw6IDANCj4+ID4+Pj4gV0FSTklORzogQXNzZXJ0aW9ucyBhcmUgZW5h
YmxlZDsgYmVuY2htYXJrcyB1bm5lY2Vzc2FyaWx5IHNsb3cNCj4+ID4+Pj4gLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQo+PiA+Pj4+IERCIHBhdGg6IFsv
dG1wL3JvY2tzZGJ0ZXN0LTAvZGJiZW5jaF0NCj4+ID4+Pj4gcmVhZHJhbmRvbWZhc3QgOiAgICAg
ICA0LjU3MCBtaWNyb3Mvb3AgMjE4ODA2IG9wcy9zZWM7ICgxMDAwMTAwIG9mDQo+PiA+Pj4+IDEw
MDAxMDAgZm91bmQsIGlzc3VlZCA0NjYzOSBub24tZXhpc3Qga2V5cykNCj4+ID4+Pj4NCj4+ID4+
Pj4gPT09PT09PT09PT09PT09PT09PT09PT09PT09DQo+PiA+Pj4+IHRoZW4gSSBtb2RpZnkgcmVh
ZHJhbmRvbWZhc3QgdG8gdXNlIEl0ZXJhdG9yIEFQSVswXToNCj4+ID4+Pj4NCj4+ID4+Pj4gW3Jv
b3RAaHVudGVyLW5vZGUyIHJvY2tzZGJdIyAuL2RiX2JlbmNoIC11c2VfZXhpc3RpbmdfZGINCj4+
ID4+Pj4gLWJlbmNobWFya3MgcmVhZHJhbmRvbWZhc3QNCj4+ID4+Pj4gTGV2ZWxEQjogICAgdmVy
c2lvbiA0LjMNCj4+ID4+Pj4gRGF0ZTogICAgICAgV2VkIEp1biAgMSAwMDozMzowMyAyMDE2DQo+
PiA+Pj4+IENQVTogICAgICAgIDMyICogSW50ZWwoUikgWGVvbihSKSBDUFUgRTUtMjY4MCAwIEAg
Mi43MEdIeg0KPj4gPj4+PiBDUFVDYWNoZTogICAyMDQ4MCBLQg0KPj4gPj4+PiBLZXlzOiAgICAg
ICAxNiBieXRlcyBlYWNoDQo+PiA+Pj4+IFZhbHVlczogICAgIDEwMCBieXRlcyBlYWNoICg1MCBi
eXRlcyBhZnRlciBjb21wcmVzc2lvbikNCj4+ID4+Pj4gRW50cmllczogICAgMTAwMDAwMA0KPj4g
Pj4+PiBQcmVmaXg6ICAgIDAgYnl0ZXMNCj4+ID4+Pj4gS2V5cyBwZXIgcHJlZml4OiAgICAwDQo+
PiA+Pj4+IFJhd1NpemU6ICAgIDExMC42IE1CIChlc3RpbWF0ZWQpDQo+PiA+Pj4+IEZpbGVTaXpl
OiAgIDYyLjkgTUIgKGVzdGltYXRlZCkNCj4+ID4+Pj4gV3JpdGVzIHBlciBzZWNvbmQ6IDANCj4+
ID4+Pj4gQ29tcHJlc3Npb246IFNuYXBweQ0KPj4gPj4+PiBNZW10YWJsZXJlcDogc2tpcF9saXN0
DQo+PiA+Pj4+IFBlcmYgTGV2ZWw6IDANCj4+ID4+Pj4gV0FSTklORzogQXNzZXJ0aW9ucyBhcmUg
ZW5hYmxlZDsgYmVuY2htYXJrcyB1bm5lY2Vzc2FyaWx5IHNsb3cNCj4+ID4+Pj4gLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQo+PiA+Pj4+IERCIHBhdGg6
IFsvdG1wL3JvY2tzZGJ0ZXN0LTAvZGJiZW5jaF0NCj4+ID4+Pj4gcmVhZHJhbmRvbWZhc3QgOiAg
ICAgIDQ1LjE4OCBtaWNyb3Mvb3AgMjIxMjkgb3BzL3NlYzsgKDEwMDAxMDAgb2YNCj4+ID4+Pj4g
MTAwMDEwMCBmb3VuZCwgaXNzdWVkIDQ2NjM5IG5vbi1leGlzdCBrZXlzKQ0KPj4gPj4+Pg0KPj4g
Pj4+Pg0KPj4gPj4+PiA0NS4xOCB1cy9vcCB2cyA0LjU3dXMvb3AhDQo+PiA+Pj4+DQo+PiA+Pj4+
IFRoZSB0ZXN0IGNhbiBiZSByZXBlYXRlZCBhbmQgZWFzeSB0byBkbyEgUGx6IGNvcnJlY3QgaWYg
SSdtIGRvaW5nDQo+PiA+Pj4+IGZvb2xpc2ggdGhpbmcgSSdtIG5vdCBhd2FyZS4uDQo+PiA+Pj4+
DQo+PiA+Pj4+IFNvIEkgcHJvcG9zYWwgdGhpcyBQUjogaHR0cHM6Ly9naXRodWIuY29tL2NlcGgv
Y2VwaC9wdWxsLzk0MTENCj4+ID4+Pj4NCj4+ID4+Pj4gV2Ugc3RpbGwgY2FuIG1ha2UgZnVydGhl
ciBpbXByb3ZlbWVudHMgYnkgc2Nhbm5pbmcgYWxsIGl0ZXJhdGUNCj4+ID4+Pj4gdXNhZ2UgdG8g
bWFrZSBpdCBiZXR0ZXIhDQo+PiA+Pj4+DQo+PiA+Pj4+IFswXToNCj4+ID4+Pj4gLS0tIGEvZGIv
ZGJfYmVuY2guY2MNCj4+ID4+Pj4gKysrIGIvZGIvZGJfYmVuY2guY2MNCj4+ID4+Pj4gQEAgLTI5
MjMsMTQgKzI5MjMsMTIgQEAgY2xhc3MgQmVuY2htYXJrIHsNCj4+ID4+Pj4gICAgICAgICAgaW50
NjRfdCBrZXlfcmFuZCA9IHRocmVhZC0+cmFuZC5OZXh0KCkgJiAocG90IC0gMSk7DQo+PiA+Pj4+
ICAgICAgICAgIEdlbmVyYXRlS2V5RnJvbUludChrZXlfcmFuZCwgRkxBR1NfbnVtLCAma2V5KTsN
Cj4+ID4+Pj4gICAgICAgICAgKytyZWFkOw0KPj4gPj4+PiAtICAgICAgICBhdXRvIHN0YXR1cyA9
IGRiLT5HZXQob3B0aW9ucywga2V5LCAmdmFsdWUpOw0KPj4gPj4+PiAtICAgICAgICBpZiAoc3Rh
dHVzLm9rKCkpIHsNCj4+ID4+Pj4gLSAgICAgICAgICArK2ZvdW5kOw0KPj4gPj4+PiAtICAgICAg
ICB9IGVsc2UgaWYgKCFzdGF0dXMuSXNOb3RGb3VuZCgpKSB7DQo+PiA+Pj4+IC0gICAgICAgICAg
ZnByaW50ZihzdGRlcnIsICJHZXQgcmV0dXJuZWQgYW4gZXJyb3I6ICVzXG4iLA0KPj4gPj4+PiAt
ICAgICAgICAgICAgICAgICAgc3RhdHVzLlRvU3RyaW5nKCkuY19zdHIoKSk7DQo+PiA+Pj4+IC0g
ICAgICAgICAgYWJvcnQoKTsNCj4+ID4+Pj4gLSAgICAgICAgfQ0KPj4gPj4+PiArICAgICAgICBJ
dGVyYXRvciogaXRlciA9IGRiLT5OZXdJdGVyYXRvcihvcHRpb25zKTsNCj4+ID4+Pj4gKyAgICAg
IGl0ZXItPlNlZWsoa2V5KTsNCj4+ID4+Pj4gKyAgICAgIGlmIChpdGVyLT5WYWxpZCgpICYmIGl0
ZXItPmtleSgpLmNvbXBhcmUoa2V5KSA9PSAwKSB7DQo+PiA+Pj4+ICsgICAgICAgIGZvdW5kKys7
DQo+PiA+Pj4+ICsgICAgICB9DQo+PiA+Pj4+ICsNCj4+ID4+Pj4gICAgICAgICAgaWYgKGtleV9y
YW5kID49IEZMQUdTX251bSkgew0KPj4gPj4+PiAgICAgICAgICAgICsrbm9uZXhpc3Q7DQo+PiA+
Pj4+ICAgICAgICAgIH0NCj4+ID4+Pj4gLS0NCj4+ID4+Pj4gVG8gdW5zdWJzY3JpYmUgZnJvbSB0
aGlzIGxpc3Q6IHNlbmQgdGhlIGxpbmUgInVuc3Vic2NyaWJlDQo+PiA+Pj4+IGNlcGgtZGV2ZWwi
IGluIHRoZSBib2R5IG9mIGEgbWVzc2FnZSB0byBtYWpvcmRvbW9Admdlci5rZXJuZWwub3JnDQo+
PiA+Pj4+IE1vcmUgbWFqb3Jkb21vIGluZm8gYXQgIGh0dHA6Ly92Z2VyLmtlcm5lbC5vcmcvbWFq
b3Jkb21vLWluZm8uaHRtbA0KPj4gPj4gLS0NCj4+ID4+IFRvIHVuc3Vic2NyaWJlIGZyb20gdGhp
cyBsaXN0OiBzZW5kIHRoZSBsaW5lICJ1bnN1YnNjcmliZSBjZXBoLWRldmVsIg0KPj4gPj4gaW4g
dGhlIGJvZHkgb2YgYSBtZXNzYWdlIHRvIG1ham9yZG9tb0B2Z2VyLmtlcm5lbC5vcmcgTW9yZQ0K
Pj4gbWFqb3Jkb21vDQo+PiA+PiBpbmZvIGF0ICBodHRwOi8vdmdlci5rZXJuZWwub3JnL21ham9y
ZG9tby1pbmZvLmh0bWwNCj4+IC0tDQo+PiBUbyB1bnN1YnNjcmliZSBmcm9tIHRoaXMgbGlzdDog
c2VuZCB0aGUgbGluZSAidW5zdWJzY3JpYmUgY2VwaC1kZXZlbCIgaW4gdGhlDQo+PiBib2R5IG9m
IGEgbWVzc2FnZSB0byBtYWpvcmRvbW9Admdlci5rZXJuZWwub3JnIE1vcmUgbWFqb3Jkb21vIGlu
Zm8gYXQNCj4+IGh0dHA6Ly92Z2VyLmtlcm5lbC5vcmcvbWFqb3Jkb21vLWluZm8uaHRtbA0K
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Varada Kari June 1, 2016, 3:37 a.m. UTC | #11
DQoNCk9uIFdlZG5lc2RheSAwMSBKdW5lIDIwMTYgMDg6NTQgQU0sIEppYW5qaWFuIEh1byB3cm90
ZToNCj4gSGkgQWxsZW4sDQo+DQo+IE9uIFR1ZSwgTWF5IDMxLCAyMDE2IGF0IDc6NTIgUE0sIEFs
bGVuIFNhbXVlbHMgPEFsbGVuLlNhbXVlbHNAc2FuZGlzay5jb20+IHdyb3RlOg0KPj4+IC0tLS0t
T3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+Pj4gRnJvbTogY2VwaC1kZXZlbC1vd25lckB2Z2VyLmtl
cm5lbC5vcmcgW21haWx0bzpjZXBoLWRldmVsLQ0KPj4+IG93bmVyQHZnZXIua2VybmVsLm9yZ10g
T24gQmVoYWxmIE9mIEhhb21haSBXYW5nDQo+Pj4gU2VudDogVHVlc2RheSwgTWF5IDMxLCAyMDE2
IDc6MzggUE0NCj4+PiBUbzogSmlhbmppYW4gSHVvIDxqaWFuamlhbi5odW9Ac2Ftc3VuZy5jb20+
DQo+Pj4gQ2M6IFNhZ2UgV2VpbCA8c3dlaWxAcmVkaGF0LmNvbT47IE1hcmsgTmVsc29uIDxtbmVs
c29uQHJlZGhhdC5jb20+Ow0KPj4+IGNlcGgtZGV2ZWxAdmdlci5rZXJuZWwub3JnDQo+Pj4gU3Vi
amVjdDogUmU6IFJvY2tzREIgSW5jb3JyZWN0IEFQSSBVc2FnZQ0KPj4+DQo+Pj4gT24gV2VkLCBK
dW4gMSwgMjAxNiBhdCAzOjQwIEFNLCBKaWFuamlhbiBIdW8gPGppYW5qaWFuLmh1b0BzYW1zdW5n
LmNvbT4NCj4+PiB3cm90ZToNCj4+Pj4gT24gVHVlLCBNYXkgMzEsIDIwMTYgYXQgMTE6MTIgQU0s
IEhhb21haSBXYW5nIDxoYW9tYWlAeHNreS5jb20+DQo+Pj4gd3JvdGU6DQo+Pj4+PiBPbiBXZWQs
IEp1biAxLCAyMDE2IGF0IDI6MDggQU0sIEppYW5qaWFuIEh1bw0KPj4+IDxqaWFuamlhbi5odW9A
c2Ftc3VuZy5jb20+IHdyb3RlOg0KPj4+Pj4+IEhpIEhhb21haSwNCj4+Pj4+Pg0KPj4+Pj4+IEkg
bm90aWNlZCB0aGlzIGFzIHdlbGwsIGFuZCBtYWRlIHNhbWUgY2hhbmdlcyB0byBSb2Nrc0RCU3Rv
cmUgaW4gdGhpcyBQUg0KPj4+IGxhc3Qgd2VlazoNCj4+Pj4+PiBodHRwczovL2dpdGh1Yi5jb20v
Y2VwaC9jZXBoL3B1bGwvOTIxNQ0KPj4+Pj4+DQo+Pj4+Pj4gT25lIHRoaW5nIHdoaWNoIGlzIGV2
ZW4gd29yc2UsICBzZWVrIHdpbGwgYnlwYXNzIHJvdyBjYWNoZSwgc28ga3YgcGFpcnMNCj4+PiB3
b24ndCBiZSBhYmxlIHRvIGJlIGNhY2hlZCBpbiByb3cgY2FjaGUuDQo+Pj4+Pj4gSSBhbSB3b3Jr
aW5nIHRvIGJlbmNobWFyayB0aGUgcGVyZm9ybWFuY2UgaW1wYWN0LCB3aWxsIHB1Ymxpc2ggdGhl
DQo+Pj4gcmVzdWx0cyBhZnRlciBJIGFtIGRvbmUgdGhpcyB3ZWVrLg0KPj4+Pj4gT2gsIGNvb2wh
IEkgdGhpbmsgeW91IGNhbiBjaGVycnktcGljayBteSBhbm90aGVyIGxldmVsZGIgZml4Lg0KPj4+
PiBObyBwcm9ibGVtLCBJIHdpbGwgZG8gdGhhdC4NCj4+Pj4+IEJUVywgZG8geW91IHBheSBhbiBh
dHRlbnRpb24gdG8gcHJlZml4IHNlZWsgYXBpPyBJIHRoaW5rIGl0IHdpbGwgYmUNCj4+Pj4+IG1v
cmUgc3VpdGFibGUgdGhhbiBjb2x1bW4gZmFtaWx5IGluIGNlcGggY2FzZS4gSWYgd2UgY2FuIGhh
dmUNCj4+Pj4+IHdlbGwtZGVmaW5lZCBwcmVmaXggcnVsZSwgd2UgY2FuIG1ha2UgbW9zdCBvZiBy
YW5nZSBxdWVyeSBjaGVhcGVyIQ0KPj4+PiBXaGVuIGtleXMgd2l0aCBzYW1lIHByZWZpeCBhcmUg
c3RvcmVkIGluIHRoZWlyIG93biBTU1QgZmlsZXMoQ0YgY2FzZSksDQo+Pj4gZXZlbiBzZWVraW5n
IHdpdGhvdXQgcHJlZml4IHdpbGwgYmUgZmFzdGVyIHRoYW4gc2Vla2luZyB3aXRoIHByZWZpeCBi
dXQgbWl4ZWQNCj4+PiB3aXRoIGRpZmZlcmVudCBwcmVmaXhlZCBrZXlzPw0KPj4+DQo+Pj4gRnJv
bSBteSBjdXJyZW50IHZpZXcsIHByZWZpeCBjb3VsZCBiZSBtb3JlIGZsZXhpYmxlLiBGb3IgZXhh
bXBsZSwgZWFjaCByZ3cNCj4+PiBidWNrZXQgaW5kZXggY291bGQgdXNlIG9uZSBwcmVmaXggdG8g
bWFrZSBlYWNoIGJ1Y2tldCBpbmRleCBvYmplY3Qgc2Vlaw0KPj4+IHNlcGFyYXRlZC4gRm9yIENG
LCBpdCB3b3VsZCBiZSB0b28gaGVhdnJ5Lg0KPj4gVGhpcyBtaWdodCBjYXVzZSBvdGhlciBwcm9i
bGVtcy4gV291bGQgaXQgZHJhbWF0aWNhbGx5IGluY3JlYXNlIHRoZSBudW1iZXIgb2YgZmlsZXMg
dGhhdCBCbHVlRlMgbmVlZHMgdG8gbWFuYWdlPyBJZiBzbywgdGhhdCBtaWdodCBlZmZlY3RpdmVs
eSBicmVhayB0aGF0IGNvZGUgdG9vIChvZiBjb3Vyc2UgaXQncyBmaXhhYmxlIGFsc28gOikpDQo+
IFRoYW5rcyBmb3IgYnJpbmdpbmcgdXAgdGhpcyBpc3N1ZS4gV2hhdCdzIGN1cnJlbnQgbGltaXQg
b2YgbnVtYmVyIG9mIGZpbGVzIGZvciBCbHVlRlMsIGZyb20geW91ciBleHBlcmllbmNlPw0KPiBK
aWFuamlhbg0KVGhlcmUgaXMgbm8gbGltaXQgcmlnaHQgbm93LCBmbm9kZSBpbm8gaXMgYSB1bml0
NjRfdC4gIERpcmVjdG9yeSBpcyBhDQpyZWYgY291bnRlZCBvYmplY3QgY29udGFpbnMgYSBmaWxl
IG1hcC4gQXMgdGhlIG51bWJlciBvZiBpbm9kZXMgZ3JvdywNCmZpbGVfbWFwIHdpbGwgZ3Jvdywg
d2lsbCBjb25zdW1lIG1vcmUgbWVtb3J5Lg0KDQpWYXJhZGENCj4+Pj4gSSBhbSBub3Qgc3VyZSB3
aGF0IG9wdGltaXphdGlvbiBwcmVmaXggc2VlayB3aWxsIHVzZSBpbnRlcm5hbGx5IGZvciBibG9j
aw0KPj4+IGJhc2VkIGZvcm1hdCwgYnV0IHRvIG1lLCBpdCdzIGhhcmQgdG8gYmVhdCB0aGUgY2Fz
ZSB3aGVuIHlvdSBvbmx5IGhhdmUgb25lDQo+Pj4gcHJlZml4ZWQga2V5cyBzdG9yZWQgc2VwYXJh
dGVseS4NCj4+Pj4+PiBKaWFuamlhbg0KPj4+Pj4+DQo+Pj4+Pj4gT24gVHVlLCBNYXkgMzEsIDIw
MTYgYXQgOTo0OSBBTSwgSGFvbWFpIFdhbmcgPGhhb21haUB4c2t5LmNvbT4NCj4+PiB3cm90ZToN
Cj4+Pj4+Pj4gSGkgU2FnZSBhbmQgTWFyaywNCj4+Pj4+Pj4NCj4+Pj4+Pj4gQXMgbWVudGlvbmVk
IGluIEJsdWVTdG9yZSBzdGFuZHVwLCBJIGZvdW5kIHJvY2tzZGIgaXRlcmF0b3IgKlNlZWsqDQo+
Pj4+Pj4+IHdvbid0IHVzZSBibG9vbSBmaWx0ZXIgbGlrZSAqR2V0Ki4NCj4+Pj4+Pj4NCj4+Pj4+
Pj4gKkdldCogaW1wbDogaXQgd2lsbCBsb29rIGF0IGZpbHRlciBmaXJzdGx5DQo+Pj4+Pj4+DQo+
Pj4gaHR0cHM6Ly9naXRodWIuY29tL2ZhY2Vib29rL3JvY2tzZGIvYmxvYi9tYXN0ZXIvdGFibGUv
YmxvY2tfYmFzZWRfdA0KPj4+Pj4+PiBhYmxlX3JlYWRlci5jYyNMMTM2OQ0KPj4+Pj4+Pg0KPj4+
Pj4+PiBJdGVyYXRvciAqU2Vlayo6IGl0IHdpbGwgZG8gYmluYXJ5IHNlYXJjaCwgYnkgZGVmYXVs
dCB3ZSBkb24ndA0KPj4+Pj4+PiBzcGVjaWZ5IHByZWZpeA0KPj4+IGZlYXR1cmUoaHR0cHM6Ly9n
aXRodWIuY29tL2ZhY2Vib29rL3JvY2tzZGIvd2lraS9QcmVmaXgtU2Vlay1BUEktDQo+Pj4gQ2hh
bmdlcykuDQo+Pj4+Pj4+IGh0dHBzOi8vZ2l0aHViLmNvbS9mYWNlYm9vay9yb2Nrc2RiL2Jsb2Iv
bWFzdGVyL3RhYmxlL2Jsb2NrLmNjI0w5NA0KPj4+Pj4+Pg0KPj4+Pj4+PiBTbyBJIHVzZSBhIHNp
bXBsZSB0ZXN0czoNCj4+Pj4+Pj4NCj4+Pj4+Pj4gLi9kYl9iZW5jaCAtbnVtIDEwMDAwMDAwICAt
YmVuY2htYXJrcyBmaWxsYmF0Y2ggZmlsbCB0aGUgZGIgZmlyc3RseQ0KPj4+Pj4+PiB3aXRoIDEw
MDB3IHJlY29yZHMuDQo+Pj4+Pj4+DQo+Pj4+Pj4+IC4vZGJfYmVuY2ggLXVzZV9leGlzdGluZ19k
YiAgLWJlbmNobWFya3MgcmVhZHJhbmRvbWZhc3QNCj4+Pj4+Pj4gcmVhZHJhbmRvbWZhc3QgY2Fz
ZSB3aWxsIHVzZSAqR2V0KiBBUEkgdG8gcmV0cml2ZSBkYXRhDQo+Pj4+Pj4+DQo+Pj4+Pj4+IFty
b290QGh1bnRlci1ub2RlMiByb2Nrc2RiXSMgLi9kYl9iZW5jaCAtdXNlX2V4aXN0aW5nX2RiDQo+
Pj4+Pj4+IC1iZW5jaG1hcmtzIHJlYWRyYW5kb21mYXN0DQo+Pj4+Pj4+DQo+Pj4+Pj4+IExldmVs
REI6ICAgIHZlcnNpb24gNC4zDQo+Pj4+Pj4+IERhdGU6ICAgICAgIFdlZCBKdW4gIDEgMDA6Mjk6
MTYgMjAxNg0KPj4+Pj4+PiBDUFU6ICAgICAgICAzMiAqIEludGVsKFIpIFhlb24oUikgQ1BVIEU1
LTI2ODAgMCBAIDIuNzBHSHoNCj4+Pj4+Pj4gQ1BVQ2FjaGU6ICAgMjA0ODAgS0INCj4+Pj4+Pj4g
S2V5czogICAgICAgMTYgYnl0ZXMgZWFjaA0KPj4+Pj4+PiBWYWx1ZXM6ICAgICAxMDAgYnl0ZXMg
ZWFjaCAoNTAgYnl0ZXMgYWZ0ZXIgY29tcHJlc3Npb24pDQo+Pj4+Pj4+IEVudHJpZXM6ICAgIDEw
MDAwMDANCj4+Pj4+Pj4gUHJlZml4OiAgICAwIGJ5dGVzDQo+Pj4+Pj4+IEtleXMgcGVyIHByZWZp
eDogICAgMA0KPj4+Pj4+PiBSYXdTaXplOiAgICAxMTAuNiBNQiAoZXN0aW1hdGVkKQ0KPj4+Pj4+
PiBGaWxlU2l6ZTogICA2Mi45IE1CIChlc3RpbWF0ZWQpDQo+Pj4+Pj4+IFdyaXRlcyBwZXIgc2Vj
b25kOiAwDQo+Pj4+Pj4+IENvbXByZXNzaW9uOiBTbmFwcHkNCj4+Pj4+Pj4gTWVtdGFibGVyZXA6
IHNraXBfbGlzdA0KPj4+Pj4+PiBQZXJmIExldmVsOiAwDQo+Pj4+Pj4+IFdBUk5JTkc6IEFzc2Vy
dGlvbnMgYXJlIGVuYWJsZWQ7IGJlbmNobWFya3MgdW5uZWNlc3NhcmlseSBzbG93DQo+Pj4+Pj4+
IC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPj4+Pj4+
PiBEQiBwYXRoOiBbL3RtcC9yb2Nrc2RidGVzdC0wL2RiYmVuY2hdDQo+Pj4+Pj4+IHJlYWRyYW5k
b21mYXN0IDogICAgICAgNC41NzAgbWljcm9zL29wIDIxODgwNiBvcHMvc2VjOyAoMTAwMDEwMCBv
Zg0KPj4+Pj4+PiAxMDAwMTAwIGZvdW5kLCBpc3N1ZWQgNDY2Mzkgbm9uLWV4aXN0IGtleXMpDQo+
Pj4+Pj4+DQo+Pj4+Pj4+ID09PT09PT09PT09PT09PT09PT09PT09PT09PQ0KPj4+Pj4+PiB0aGVu
IEkgbW9kaWZ5IHJlYWRyYW5kb21mYXN0IHRvIHVzZSBJdGVyYXRvciBBUElbMF06DQo+Pj4+Pj4+
DQo+Pj4+Pj4+IFtyb290QGh1bnRlci1ub2RlMiByb2Nrc2RiXSMgLi9kYl9iZW5jaCAtdXNlX2V4
aXN0aW5nX2RiDQo+Pj4+Pj4+IC1iZW5jaG1hcmtzIHJlYWRyYW5kb21mYXN0DQo+Pj4+Pj4+IExl
dmVsREI6ICAgIHZlcnNpb24gNC4zDQo+Pj4+Pj4+IERhdGU6ICAgICAgIFdlZCBKdW4gIDEgMDA6
MzM6MDMgMjAxNg0KPj4+Pj4+PiBDUFU6ICAgICAgICAzMiAqIEludGVsKFIpIFhlb24oUikgQ1BV
IEU1LTI2ODAgMCBAIDIuNzBHSHoNCj4+Pj4+Pj4gQ1BVQ2FjaGU6ICAgMjA0ODAgS0INCj4+Pj4+
Pj4gS2V5czogICAgICAgMTYgYnl0ZXMgZWFjaA0KPj4+Pj4+PiBWYWx1ZXM6ICAgICAxMDAgYnl0
ZXMgZWFjaCAoNTAgYnl0ZXMgYWZ0ZXIgY29tcHJlc3Npb24pDQo+Pj4+Pj4+IEVudHJpZXM6ICAg
IDEwMDAwMDANCj4+Pj4+Pj4gUHJlZml4OiAgICAwIGJ5dGVzDQo+Pj4+Pj4+IEtleXMgcGVyIHBy
ZWZpeDogICAgMA0KPj4+Pj4+PiBSYXdTaXplOiAgICAxMTAuNiBNQiAoZXN0aW1hdGVkKQ0KPj4+
Pj4+PiBGaWxlU2l6ZTogICA2Mi45IE1CIChlc3RpbWF0ZWQpDQo+Pj4+Pj4+IFdyaXRlcyBwZXIg
c2Vjb25kOiAwDQo+Pj4+Pj4+IENvbXByZXNzaW9uOiBTbmFwcHkNCj4+Pj4+Pj4gTWVtdGFibGVy
ZXA6IHNraXBfbGlzdA0KPj4+Pj4+PiBQZXJmIExldmVsOiAwDQo+Pj4+Pj4+IFdBUk5JTkc6IEFz
c2VydGlvbnMgYXJlIGVuYWJsZWQ7IGJlbmNobWFya3MgdW5uZWNlc3NhcmlseSBzbG93DQo+Pj4+
Pj4+IC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPj4+
Pj4+PiBEQiBwYXRoOiBbL3RtcC9yb2Nrc2RidGVzdC0wL2RiYmVuY2hdDQo+Pj4+Pj4+IHJlYWRy
YW5kb21mYXN0IDogICAgICA0NS4xODggbWljcm9zL29wIDIyMTI5IG9wcy9zZWM7ICgxMDAwMTAw
IG9mDQo+Pj4+Pj4+IDEwMDAxMDAgZm91bmQsIGlzc3VlZCA0NjYzOSBub24tZXhpc3Qga2V5cykN
Cj4+Pj4+Pj4NCj4+Pj4+Pj4NCj4+Pj4+Pj4gNDUuMTggdXMvb3AgdnMgNC41N3VzL29wIQ0KPj4+
Pj4+Pg0KPj4+Pj4+PiBUaGUgdGVzdCBjYW4gYmUgcmVwZWF0ZWQgYW5kIGVhc3kgdG8gZG8hIFBs
eiBjb3JyZWN0IGlmIEknbSBkb2luZw0KPj4+Pj4+PiBmb29saXNoIHRoaW5nIEknbSBub3QgYXdh
cmUuLg0KPj4+Pj4+Pg0KPj4+Pj4+PiBTbyBJIHByb3Bvc2FsIHRoaXMgUFI6IGh0dHBzOi8vZ2l0
aHViLmNvbS9jZXBoL2NlcGgvcHVsbC85NDExDQo+Pj4+Pj4+DQo+Pj4+Pj4+IFdlIHN0aWxsIGNh
biBtYWtlIGZ1cnRoZXIgaW1wcm92ZW1lbnRzIGJ5IHNjYW5uaW5nIGFsbCBpdGVyYXRlDQo+Pj4+
Pj4+IHVzYWdlIHRvIG1ha2UgaXQgYmV0dGVyIQ0KPj4+Pj4+Pg0KPj4+Pj4+PiBbMF06DQo+Pj4+
Pj4+IC0tLSBhL2RiL2RiX2JlbmNoLmNjDQo+Pj4+Pj4+ICsrKyBiL2RiL2RiX2JlbmNoLmNjDQo+
Pj4+Pj4+IEBAIC0yOTIzLDE0ICsyOTIzLDEyIEBAIGNsYXNzIEJlbmNobWFyayB7DQo+Pj4+Pj4+
ICAgICAgICAgIGludDY0X3Qga2V5X3JhbmQgPSB0aHJlYWQtPnJhbmQuTmV4dCgpICYgKHBvdCAt
IDEpOw0KPj4+Pj4+PiAgICAgICAgICBHZW5lcmF0ZUtleUZyb21JbnQoa2V5X3JhbmQsIEZMQUdT
X251bSwgJmtleSk7DQo+Pj4+Pj4+ICAgICAgICAgICsrcmVhZDsNCj4+Pj4+Pj4gLSAgICAgICAg
YXV0byBzdGF0dXMgPSBkYi0+R2V0KG9wdGlvbnMsIGtleSwgJnZhbHVlKTsNCj4+Pj4+Pj4gLSAg
ICAgICAgaWYgKHN0YXR1cy5vaygpKSB7DQo+Pj4+Pj4+IC0gICAgICAgICAgKytmb3VuZDsNCj4+
Pj4+Pj4gLSAgICAgICAgfSBlbHNlIGlmICghc3RhdHVzLklzTm90Rm91bmQoKSkgew0KPj4+Pj4+
PiAtICAgICAgICAgIGZwcmludGYoc3RkZXJyLCAiR2V0IHJldHVybmVkIGFuIGVycm9yOiAlc1xu
IiwNCj4+Pj4+Pj4gLSAgICAgICAgICAgICAgICAgIHN0YXR1cy5Ub1N0cmluZygpLmNfc3RyKCkp
Ow0KPj4+Pj4+PiAtICAgICAgICAgIGFib3J0KCk7DQo+Pj4+Pj4+IC0gICAgICAgIH0NCj4+Pj4+
Pj4gKyAgICAgICAgSXRlcmF0b3IqIGl0ZXIgPSBkYi0+TmV3SXRlcmF0b3Iob3B0aW9ucyk7DQo+
Pj4+Pj4+ICsgICAgICBpdGVyLT5TZWVrKGtleSk7DQo+Pj4+Pj4+ICsgICAgICBpZiAoaXRlci0+
VmFsaWQoKSAmJiBpdGVyLT5rZXkoKS5jb21wYXJlKGtleSkgPT0gMCkgew0KPj4+Pj4+PiArICAg
ICAgICBmb3VuZCsrOw0KPj4+Pj4+PiArICAgICAgfQ0KPj4+Pj4+PiArDQo+Pj4+Pj4+ICAgICAg
ICAgIGlmIChrZXlfcmFuZCA+PSBGTEFHU19udW0pIHsNCj4+Pj4+Pj4gICAgICAgICAgICArK25v
bmV4aXN0Ow0KPj4+Pj4+PiAgICAgICAgICB9DQo+Pj4+Pj4+IC0tDQo+Pj4+Pj4+IFRvIHVuc3Vi
c2NyaWJlIGZyb20gdGhpcyBsaXN0OiBzZW5kIHRoZSBsaW5lICJ1bnN1YnNjcmliZQ0KPj4+Pj4+
PiBjZXBoLWRldmVsIiBpbiB0aGUgYm9keSBvZiBhIG1lc3NhZ2UgdG8gbWFqb3Jkb21vQHZnZXIu
a2VybmVsLm9yZw0KPj4+Pj4+PiBNb3JlIG1ham9yZG9tbyBpbmZvIGF0ICBodHRwOi8vdmdlci5r
ZXJuZWwub3JnL21ham9yZG9tby1pbmZvLmh0bWwNCj4+Pj4+IC0tDQo+Pj4+PiBUbyB1bnN1YnNj
cmliZSBmcm9tIHRoaXMgbGlzdDogc2VuZCB0aGUgbGluZSAidW5zdWJzY3JpYmUgY2VwaC1kZXZl
bCINCj4+Pj4+IGluIHRoZSBib2R5IG9mIGEgbWVzc2FnZSB0byBtYWpvcmRvbW9Admdlci5rZXJu
ZWwub3JnIE1vcmUNCj4+PiBtYWpvcmRvbW8NCj4+Pj4+IGluZm8gYXQgIGh0dHA6Ly92Z2VyLmtl
cm5lbC5vcmcvbWFqb3Jkb21vLWluZm8uaHRtbA0KPj4+IC0tDQo+Pj4gVG8gdW5zdWJzY3JpYmUg
ZnJvbSB0aGlzIGxpc3Q6IHNlbmQgdGhlIGxpbmUgInVuc3Vic2NyaWJlIGNlcGgtZGV2ZWwiIGlu
IHRoZQ0KPj4+IGJvZHkgb2YgYSBtZXNzYWdlIHRvIG1ham9yZG9tb0B2Z2VyLmtlcm5lbC5vcmcg
TW9yZSBtYWpvcmRvbW8gaW5mbyBhdA0KPj4+IGh0dHA6Ly92Z2VyLmtlcm5lbC5vcmcvbWFqb3Jk
b21vLWluZm8uaHRtbA0KPiBO77+977+977+977+977+9cu+/ve+/vXnvv73vv73vv71i77+9WO+/
ve+/vcendu+/vV7vv70p3rp7Lm7vv70r77+977+977+9eu+/vV1677+977+977+9e2F577+9yofa
me+/vSxq77+977+9Zu+/ve+/ve+/vWjvv73vv73vv71677+977+9d++/ve+/ve+/ve+/ve+/ve+/
vWo6K3bvv73vv73vv71377+9au+/vW3vv73vv73vv73vv73vv73vv73vv73vv716Wivvv73vv73v
v73vv73vv73domoi77+977+9Ie+/vWkNCg0KUExFQVNFIE5PVEU6IFRoZSBpbmZvcm1hdGlvbiBj
b250YWluZWQgaW4gdGhpcyBlbGVjdHJvbmljIG1haWwgbWVzc2FnZSBpcyBpbnRlbmRlZCBvbmx5
IGZvciB0aGUgdXNlIG9mIHRoZSBkZXNpZ25hdGVkIHJlY2lwaWVudChzKSBuYW1lZCBhYm92ZS4g
SWYgdGhlIHJlYWRlciBvZiB0aGlzIG1lc3NhZ2UgaXMgbm90IHRoZSBpbnRlbmRlZCByZWNpcGll
bnQsIHlvdSBhcmUgaGVyZWJ5IG5vdGlmaWVkIHRoYXQgeW91IGhhdmUgcmVjZWl2ZWQgdGhpcyBt
ZXNzYWdlIGluIGVycm9yIGFuZCB0aGF0IGFueSByZXZpZXcsIGRpc3NlbWluYXRpb24sIGRpc3Ry
aWJ1dGlvbiwgb3IgY29weWluZyBvZiB0aGlzIG1lc3NhZ2UgaXMgc3RyaWN0bHkgcHJvaGliaXRl
ZC4gSWYgeW91IGhhdmUgcmVjZWl2ZWQgdGhpcyBjb21tdW5pY2F0aW9uIGluIGVycm9yLCBwbGVh
c2Ugbm90aWZ5IHRoZSBzZW5kZXIgYnkgdGVsZXBob25lIG9yIGUtbWFpbCAoYXMgc2hvd24gYWJv
dmUpIGltbWVkaWF0ZWx5IGFuZCBkZXN0cm95IGFueSBhbmQgYWxsIGNvcGllcyBvZiB0aGlzIG1l
c3NhZ2UgaW4geW91ciBwb3NzZXNzaW9uICh3aGV0aGVyIGhhcmQgY29waWVzIG9yIGVsZWN0cm9u
aWNhbGx5IHN0b3JlZCBjb3BpZXMpLg0K
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Allen Samuels June 1, 2016, 1:46 p.m. UTC | #12
> -----Original Message-----

> From: Varada Kari

> Sent: Tuesday, May 31, 2016 8:38 PM

> To: Jianjian Huo <jianjian.huo@samsung.com>; Allen Samuels

> <Allen.Samuels@sandisk.com>; Haomai Wang <haomai@xsky.com>

> Cc: Sage Weil <sweil@redhat.com>; Mark Nelson <mnelson@redhat.com>;

> ceph-devel@vger.kernel.org

> Subject: Re: RocksDB Incorrect API Usage

> 

> 

> 

> On Wednesday 01 June 2016 08:54 AM, Jianjian Huo wrote:

> > Hi Allen,

> >

> > On Tue, May 31, 2016 at 7:52 PM, Allen Samuels

> <Allen.Samuels@sandisk.com> wrote:

> >>> -----Original Message-----

> >>> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-

> >>> owner@vger.kernel.org] On Behalf Of Haomai Wang

> >>> Sent: Tuesday, May 31, 2016 7:38 PM

> >>> To: Jianjian Huo <jianjian.huo@samsung.com>

> >>> Cc: Sage Weil <sweil@redhat.com>; Mark Nelson

> <mnelson@redhat.com>;

> >>> ceph-devel@vger.kernel.org

> >>> Subject: Re: RocksDB Incorrect API Usage

> >>>

> >>> On Wed, Jun 1, 2016 at 3:40 AM, Jianjian Huo

> >>> <jianjian.huo@samsung.com>

> >>> wrote:

> >>>> On Tue, May 31, 2016 at 11:12 AM, Haomai Wang <haomai@xsky.com>

> >>> wrote:

> >>>>> On Wed, Jun 1, 2016 at 2:08 AM, Jianjian Huo

> >>> <jianjian.huo@samsung.com> wrote:

> >>>>>> Hi Haomai,

> >>>>>>

> >>>>>> I noticed this as well, and made same changes to RocksDBStore in

> >>>>>> this PR

> >>> last week:

> >>>>>> https://github.com/ceph/ceph/pull/9215

> >>>>>>

> >>>>>> One thing which is even worse,  seek will bypass row cache, so kv

> >>>>>> pairs

> >>> won't be able to be cached in row cache.

> >>>>>> I am working to benchmark the performance impact, will publish

> >>>>>> the

> >>> results after I am done this week.

> >>>>> Oh, cool! I think you can cherry-pick my another leveldb fix.

> >>>> No problem, I will do that.

> >>>>> BTW, do you pay an attention to prefix seek api? I think it will

> >>>>> be more suitable than column family in ceph case. If we can have

> >>>>> well-defined prefix rule, we can make most of range query cheaper!

> >>>> When keys with same prefix are stored in their own SST files(CF

> >>>> case),

> >>> even seeking without prefix will be faster than seeking with prefix

> >>> but mixed with different prefixed keys?

> >>>

> >>> From my current view, prefix could be more flexible. For example,

> >>> each rgw bucket index could use one prefix to make each bucket index

> >>> object seek separated. For CF, it would be too heavry.

> >> This might cause other problems. Would it dramatically increase the

> >> number of files that BlueFS needs to manage? If so, that might

> >> effectively break that code too (of course it's fixable also :))

> > Thanks for bringing up this issue. What's current limit of number of files for

> BlueFS, from your experience?

> > Jianjian

> There is no limit right now, fnode ino is a unit64_t.  Directory is a ref counted

> object contains a file map. As the number of inodes grow, file_map will grow,

> will consume more memory.

> 

> Varada


Technically correct, my concern is one of performance for BlueFS journal compaction.
 
> >>>> I am not sure what optimization prefix seek will use internally for

> >>>> block

> >>> based format, but to me, it's hard to beat the case when you only

> >>> have one prefixed keys stored separately.

> >>>>>> Jianjian

> >>>>>>

> >>>>>> On Tue, May 31, 2016 at 9:49 AM, Haomai Wang

> <haomai@xsky.com>

> >>> wrote:

> >>>>>>> Hi Sage and Mark,

> >>>>>>>

> >>>>>>> As mentioned in BlueStore standup, I found rocksdb iterator

> >>>>>>> *Seek* won't use bloom filter like *Get*.

> >>>>>>>

> >>>>>>> *Get* impl: it will look at filter firstly

> >>>>>>>

> >>>

> https://github.com/facebook/rocksdb/blob/master/table/block_based_t

> >>>>>>> able_reader.cc#L1369

> >>>>>>>

> >>>>>>> Iterator *Seek*: it will do binary search, by default we don't

> >>>>>>> specify prefix

> >>> feature(https://github.com/facebook/rocksdb/wiki/Prefix-Seek-API-

> >>> Changes).

> >>>>>>>

> https://github.com/facebook/rocksdb/blob/master/table/block.cc#L

> >>>>>>> 94

> >>>>>>>

> >>>>>>> So I use a simple tests:

> >>>>>>>

> >>>>>>> ./db_bench -num 10000000  -benchmarks fillbatch fill the db

> >>>>>>> firstly with 1000w records.

> >>>>>>>

> >>>>>>> ./db_bench -use_existing_db  -benchmarks readrandomfast

> >>>>>>> readrandomfast case will use *Get* API to retrive data

> >>>>>>>

> >>>>>>> [root@hunter-node2 rocksdb]# ./db_bench -use_existing_db

> >>>>>>> -benchmarks readrandomfast

> >>>>>>>

> >>>>>>> LevelDB:    version 4.3

> >>>>>>> Date:       Wed Jun  1 00:29:16 2016

> >>>>>>> CPU:        32 * Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz

> >>>>>>> CPUCache:   20480 KB

> >>>>>>> Keys:       16 bytes each

> >>>>>>> Values:     100 bytes each (50 bytes after compression)

> >>>>>>> Entries:    1000000

> >>>>>>> Prefix:    0 bytes

> >>>>>>> Keys per prefix:    0

> >>>>>>> RawSize:    110.6 MB (estimated)

> >>>>>>> FileSize:   62.9 MB (estimated)

> >>>>>>> Writes per second: 0

> >>>>>>> Compression: Snappy

> >>>>>>> Memtablerep: skip_list

> >>>>>>> Perf Level: 0

> >>>>>>> WARNING: Assertions are enabled; benchmarks unnecessarily slow

> >>>>>>> ------------------------------------------------

> >>>>>>> DB path: [/tmp/rocksdbtest-0/dbbench]

> >>>>>>> readrandomfast :       4.570 micros/op 218806 ops/sec; (1000100 of

> >>>>>>> 1000100 found, issued 46639 non-exist keys)

> >>>>>>>

> >>>>>>> ===========================

> >>>>>>> then I modify readrandomfast to use Iterator API[0]:

> >>>>>>>

> >>>>>>> [root@hunter-node2 rocksdb]# ./db_bench -use_existing_db

> >>>>>>> -benchmarks readrandomfast

> >>>>>>> LevelDB:    version 4.3

> >>>>>>> Date:       Wed Jun  1 00:33:03 2016

> >>>>>>> CPU:        32 * Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz

> >>>>>>> CPUCache:   20480 KB

> >>>>>>> Keys:       16 bytes each

> >>>>>>> Values:     100 bytes each (50 bytes after compression)

> >>>>>>> Entries:    1000000

> >>>>>>> Prefix:    0 bytes

> >>>>>>> Keys per prefix:    0

> >>>>>>> RawSize:    110.6 MB (estimated)

> >>>>>>> FileSize:   62.9 MB (estimated)

> >>>>>>> Writes per second: 0

> >>>>>>> Compression: Snappy

> >>>>>>> Memtablerep: skip_list

> >>>>>>> Perf Level: 0

> >>>>>>> WARNING: Assertions are enabled; benchmarks unnecessarily slow

> >>>>>>> ------------------------------------------------

> >>>>>>> DB path: [/tmp/rocksdbtest-0/dbbench]

> >>>>>>> readrandomfast :      45.188 micros/op 22129 ops/sec; (1000100 of

> >>>>>>> 1000100 found, issued 46639 non-exist keys)

> >>>>>>>

> >>>>>>>

> >>>>>>> 45.18 us/op vs 4.57us/op!

> >>>>>>>

> >>>>>>> The test can be repeated and easy to do! Plz correct if I'm

> >>>>>>> doing foolish thing I'm not aware..

> >>>>>>>

> >>>>>>> So I proposal this PR: https://github.com/ceph/ceph/pull/9411

> >>>>>>>

> >>>>>>> We still can make further improvements by scanning all iterate

> >>>>>>> usage to make it better!

> >>>>>>>

> >>>>>>> [0]:

> >>>>>>> --- a/db/db_bench.cc

> >>>>>>> +++ b/db/db_bench.cc

> >>>>>>> @@ -2923,14 +2923,12 @@ class Benchmark {

> >>>>>>>          int64_t key_rand = thread->rand.Next() & (pot - 1);

> >>>>>>>          GenerateKeyFromInt(key_rand, FLAGS_num, &key);

> >>>>>>>          ++read;

> >>>>>>> -        auto status = db->Get(options, key, &value);

> >>>>>>> -        if (status.ok()) {

> >>>>>>> -          ++found;

> >>>>>>> -        } else if (!status.IsNotFound()) {

> >>>>>>> -          fprintf(stderr, "Get returned an error: %s\n",

> >>>>>>> -                  status.ToString().c_str());

> >>>>>>> -          abort();

> >>>>>>> -        }

> >>>>>>> +        Iterator* iter = db->NewIterator(options);

> >>>>>>> +      iter->Seek(key);

> >>>>>>> +      if (iter->Valid() && iter->key().compare(key) == 0) {

> >>>>>>> +        found++;

> >>>>>>> +      }

> >>>>>>> +

> >>>>>>>          if (key_rand >= FLAGS_num) {

> >>>>>>>            ++nonexist;

> >>>>>>>          }

> >>>>>>> --

> >>>>>>> To unsubscribe from this list: send the line "unsubscribe

> >>>>>>> ceph-devel" in the body of a message to

> >>>>>>> majordomo@vger.kernel.org More majordomo info at

> >>>>>>> http://vger.kernel.org/majordomo-info.html

> >>>>> --

> >>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"

> >>>>> in the body of a message to majordomo@vger.kernel.org More

> >>> majordomo

> >>>>> info at  http://vger.kernel.org/majordomo-info.html

> >>> --

> >>> To unsubscribe from this list: send the line "unsubscribe

> >>> ceph-devel" in the body of a message to majordomo@vger.kernel.org

> >>> More majordomo info at http://vger.kernel.org/majordomo-info.html

> > N     r  y   b X  ǧv ^ )޺{.n +   z ]z   {ay ʇڙ ,j  f   h   z  w      j

> > :+v   w j m        zZ+     ݢj"  ! i
Varada Kari June 1, 2016, 2:24 p.m. UTC | #13
On Wednesday 01 June 2016 07:16 PM, Allen Samuels wrote:
>> -----Original Message-----

>> From: Varada Kari

>> Sent: Tuesday, May 31, 2016 8:38 PM

>> To: Jianjian Huo <jianjian.huo@samsung.com>; Allen Samuels

>> <Allen.Samuels@sandisk.com>; Haomai Wang <haomai@xsky.com>

>> Cc: Sage Weil <sweil@redhat.com>; Mark Nelson <mnelson@redhat.com>;

>> ceph-devel@vger.kernel.org

>> Subject: Re: RocksDB Incorrect API Usage

>>

>>

>>

>> On Wednesday 01 June 2016 08:54 AM, Jianjian Huo wrote:

>>> Hi Allen,

>>>

>>> On Tue, May 31, 2016 at 7:52 PM, Allen Samuels

>> <Allen.Samuels@sandisk.com> wrote:

>>>>> -----Original Message-----

>>>>> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-

>>>>> owner@vger.kernel.org] On Behalf Of Haomai Wang

>>>>> Sent: Tuesday, May 31, 2016 7:38 PM

>>>>> To: Jianjian Huo <jianjian.huo@samsung.com>

>>>>> Cc: Sage Weil <sweil@redhat.com>; Mark Nelson

>> <mnelson@redhat.com>;

>>>>> ceph-devel@vger.kernel.org

>>>>> Subject: Re: RocksDB Incorrect API Usage

>>>>>

>>>>> On Wed, Jun 1, 2016 at 3:40 AM, Jianjian Huo

>>>>> <jianjian.huo@samsung.com>

>>>>> wrote:

>>>>>> On Tue, May 31, 2016 at 11:12 AM, Haomai Wang <haomai@xsky.com>

>>>>> wrote:

>>>>>>> On Wed, Jun 1, 2016 at 2:08 AM, Jianjian Huo

>>>>> <jianjian.huo@samsung.com> wrote:

>>>>>>>> Hi Haomai,

>>>>>>>>

>>>>>>>> I noticed this as well, and made same changes to RocksDBStore in

>>>>>>>> this PR

>>>>> last week:

>>>>>>>> https://github.com/ceph/ceph/pull/9215

>>>>>>>>

>>>>>>>> One thing which is even worse,  seek will bypass row cache, so kv

>>>>>>>> pairs

>>>>> won't be able to be cached in row cache.

>>>>>>>> I am working to benchmark the performance impact, will publish

>>>>>>>> the

>>>>> results after I am done this week.

>>>>>>> Oh, cool! I think you can cherry-pick my another leveldb fix.

>>>>>> No problem, I will do that.

>>>>>>> BTW, do you pay an attention to prefix seek api? I think it will

>>>>>>> be more suitable than column family in ceph case. If we can have

>>>>>>> well-defined prefix rule, we can make most of range query cheaper!

>>>>>> When keys with same prefix are stored in their own SST files(CF

>>>>>> case),

>>>>> even seeking without prefix will be faster than seeking with prefix

>>>>> but mixed with different prefixed keys?

>>>>>

>>>>> From my current view, prefix could be more flexible. For example,

>>>>> each rgw bucket index could use one prefix to make each bucket index

>>>>> object seek separated. For CF, it would be too heavry.

>>>> This might cause other problems. Would it dramatically increase the

>>>> number of files that BlueFS needs to manage? If so, that might

>>>> effectively break that code too (of course it's fixable also :))

>>> Thanks for bringing up this issue. What's current limit of number of files for

>> BlueFS, from your experience?

>>> Jianjian

>> There is no limit right now, fnode ino is a unit64_t.  Directory is a ref counted

>> object contains a file map. As the number of inodes grow, file_map will grow,

>> will consume more memory.

>>

>> Varada

> Technically correct, my concern is one of performance for BlueFS journal compaction.

>

Yes. I am working on optimizing the latency part during the compaction.

Varada
>>>>>> I am not sure what optimization prefix seek will use internally for

>>>>>> block

>>>>> based format, but to me, it's hard to beat the case when you only

>>>>> have one prefixed keys stored separately.

>>>>>>>> Jianjian

>>>>>>>>

>>>>>>>> On Tue, May 31, 2016 at 9:49 AM, Haomai Wang

>> <haomai@xsky.com>

>>>>> wrote:

>>>>>>>>> Hi Sage and Mark,

>>>>>>>>>

>>>>>>>>> As mentioned in BlueStore standup, I found rocksdb iterator

>>>>>>>>> *Seek* won't use bloom filter like *Get*.

>>>>>>>>>

>>>>>>>>> *Get* impl: it will look at filter firstly

>>>>>>>>>

>> https://github.com/facebook/rocksdb/blob/master/table/block_based_t

>>>>>>>>> able_reader.cc#L1369

>>>>>>>>>

>>>>>>>>> Iterator *Seek*: it will do binary search, by default we don't

>>>>>>>>> specify prefix

>>>>> feature(https://github.com/facebook/rocksdb/wiki/Prefix-Seek-API-

>>>>> Changes).

>> https://github.com/facebook/rocksdb/blob/master/table/block.cc#L

>>>>>>>>> 94

>>>>>>>>>

>>>>>>>>> So I use a simple tests:

>>>>>>>>>

>>>>>>>>> ./db_bench -num 10000000  -benchmarks fillbatch fill the db

>>>>>>>>> firstly with 1000w records.

>>>>>>>>>

>>>>>>>>> ./db_bench -use_existing_db  -benchmarks readrandomfast

>>>>>>>>> readrandomfast case will use *Get* API to retrive data

>>>>>>>>>

>>>>>>>>> [root@hunter-node2 rocksdb]# ./db_bench -use_existing_db

>>>>>>>>> -benchmarks readrandomfast

>>>>>>>>>

>>>>>>>>> LevelDB:    version 4.3

>>>>>>>>> Date:       Wed Jun  1 00:29:16 2016

>>>>>>>>> CPU:        32 * Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz

>>>>>>>>> CPUCache:   20480 KB

>>>>>>>>> Keys:       16 bytes each

>>>>>>>>> Values:     100 bytes each (50 bytes after compression)

>>>>>>>>> Entries:    1000000

>>>>>>>>> Prefix:    0 bytes

>>>>>>>>> Keys per prefix:    0

>>>>>>>>> RawSize:    110.6 MB (estimated)

>>>>>>>>> FileSize:   62.9 MB (estimated)

>>>>>>>>> Writes per second: 0

>>>>>>>>> Compression: Snappy

>>>>>>>>> Memtablerep: skip_list

>>>>>>>>> Perf Level: 0

>>>>>>>>> WARNING: Assertions are enabled; benchmarks unnecessarily slow

>>>>>>>>> ------------------------------------------------

>>>>>>>>> DB path: [/tmp/rocksdbtest-0/dbbench]

>>>>>>>>> readrandomfast :       4.570 micros/op 218806 ops/sec; (1000100 of

>>>>>>>>> 1000100 found, issued 46639 non-exist keys)

>>>>>>>>>

>>>>>>>>> ===========================

>>>>>>>>> then I modify readrandomfast to use Iterator API[0]:

>>>>>>>>>

>>>>>>>>> [root@hunter-node2 rocksdb]# ./db_bench -use_existing_db

>>>>>>>>> -benchmarks readrandomfast

>>>>>>>>> LevelDB:    version 4.3

>>>>>>>>> Date:       Wed Jun  1 00:33:03 2016

>>>>>>>>> CPU:        32 * Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz

>>>>>>>>> CPUCache:   20480 KB

>>>>>>>>> Keys:       16 bytes each

>>>>>>>>> Values:     100 bytes each (50 bytes after compression)

>>>>>>>>> Entries:    1000000

>>>>>>>>> Prefix:    0 bytes

>>>>>>>>> Keys per prefix:    0

>>>>>>>>> RawSize:    110.6 MB (estimated)

>>>>>>>>> FileSize:   62.9 MB (estimated)

>>>>>>>>> Writes per second: 0

>>>>>>>>> Compression: Snappy

>>>>>>>>> Memtablerep: skip_list

>>>>>>>>> Perf Level: 0

>>>>>>>>> WARNING: Assertions are enabled; benchmarks unnecessarily slow

>>>>>>>>> ------------------------------------------------

>>>>>>>>> DB path: [/tmp/rocksdbtest-0/dbbench]

>>>>>>>>> readrandomfast :      45.188 micros/op 22129 ops/sec; (1000100 of

>>>>>>>>> 1000100 found, issued 46639 non-exist keys)

>>>>>>>>>

>>>>>>>>>

>>>>>>>>> 45.18 us/op vs 4.57us/op!

>>>>>>>>>

>>>>>>>>> The test can be repeated and easy to do! Plz correct if I'm

>>>>>>>>> doing foolish thing I'm not aware..

>>>>>>>>>

>>>>>>>>> So I proposal this PR: https://github.com/ceph/ceph/pull/9411

>>>>>>>>>

>>>>>>>>> We still can make further improvements by scanning all iterate

>>>>>>>>> usage to make it better!

>>>>>>>>>

>>>>>>>>> [0]:

>>>>>>>>> --- a/db/db_bench.cc

>>>>>>>>> +++ b/db/db_bench.cc

>>>>>>>>> @@ -2923,14 +2923,12 @@ class Benchmark {

>>>>>>>>>          int64_t key_rand = thread->rand.Next() & (pot - 1);

>>>>>>>>>          GenerateKeyFromInt(key_rand, FLAGS_num, &key);

>>>>>>>>>          ++read;

>>>>>>>>> -        auto status = db->Get(options, key, &value);

>>>>>>>>> -        if (status.ok()) {

>>>>>>>>> -          ++found;

>>>>>>>>> -        } else if (!status.IsNotFound()) {

>>>>>>>>> -          fprintf(stderr, "Get returned an error: %s\n",

>>>>>>>>> -                  status.ToString().c_str());

>>>>>>>>> -          abort();

>>>>>>>>> -        }

>>>>>>>>> +        Iterator* iter = db->NewIterator(options);

>>>>>>>>> +      iter->Seek(key);

>>>>>>>>> +      if (iter->Valid() && iter->key().compare(key) == 0) {

>>>>>>>>> +        found++;

>>>>>>>>> +      }

>>>>>>>>> +

>>>>>>>>>          if (key_rand >= FLAGS_num) {

>>>>>>>>>            ++nonexist;

>>>>>>>>>          }

>>>>>>>>> --

>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe

>>>>>>>>> ceph-devel" in the body of a message to

>>>>>>>>> majordomo@vger.kernel.org More majordomo info at

>>>>>>>>> http://vger.kernel.org/majordomo-info.html

>>>>>>> --

>>>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"

>>>>>>> in the body of a message to majordomo@vger.kernel.org More

>>>>> majordomo

>>>>>>> info at  http://vger.kernel.org/majordomo-info.html

>>>>> --

>>>>> To unsubscribe from this list: send the line "unsubscribe

>>>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org

>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html

>>> N     r  y   b X  ǧv ^ )޺{.n +   z ]z   {ay ʇڙ ,j  f   h   z  w      j

>>> :+v   w j m        zZ+     ݢj"  ! i

>


PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

Patch
diff mbox

===========================
then I modify readrandomfast to use Iterator API[0]:

[root@hunter-node2 rocksdb]# ./db_bench -use_existing_db  -benchmarks
readrandomfast
LevelDB:    version 4.3
Date:       Wed Jun  1 00:33:03 2016
CPU:        32 * Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
CPUCache:   20480 KB
Keys:       16 bytes each
Values:     100 bytes each (50 bytes after compression)
Entries:    1000000
Prefix:    0 bytes
Keys per prefix:    0
RawSize:    110.6 MB (estimated)
FileSize:   62.9 MB (estimated)
Writes per second: 0
Compression: Snappy
Memtablerep: skip_list
Perf Level: 0
WARNING: Assertions are enabled; benchmarks unnecessarily slow
------------------------------------------------
DB path: [/tmp/rocksdbtest-0/dbbench]
readrandomfast :      45.188 micros/op 22129 ops/sec; (1000100 of
1000100 found, issued 46639 non-exist keys)


45.18 us/op vs 4.57us/op!

The test can be repeated and easy to do! Plz correct if I'm doing
foolish thing I'm not aware..

So I proposal this PR: https://github.com/ceph/ceph/pull/9411

We still can make further improvements by scanning all iterate usage
to make it better!

[0]:
--- a/db/db_bench.cc
+++ b/db/db_bench.cc
@@ -2923,14 +2923,12 @@  class Benchmark {
         int64_t key_rand = thread->rand.Next() & (pot - 1);
         GenerateKeyFromInt(key_rand, FLAGS_num, &key);
         ++read;
-        auto status = db->Get(options, key, &value);
-        if (status.ok()) {
-          ++found;
-        } else if (!status.IsNotFound()) {
-          fprintf(stderr, "Get returned an error: %s\n",
-                  status.ToString().c_str());
-          abort();
-        }
+        Iterator* iter = db->NewIterator(options);
+      iter->Seek(key);
+      if (iter->Valid() && iter->key().compare(key) == 0) {
+        found++;
+      }
+
         if (key_rand >= FLAGS_num) {
           ++nonexist;
         }