| Message ID | CACJqLyYtHkeQ+jSNi7JPJzRb6wfCcNAuCSr_YyTwdqA_kzzP_g@mail.gmail.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On 05/31/2016 11:49 AM, Haomai Wang wrote: > Hi Sage and Mark, > > As mentioned in BlueStore standup, I found rocksdb iterator *Seek* > won't use bloom filter like *Get*. > > *Get* impl: it will look at filter firstly > https://github.com/facebook/rocksdb/blob/master/table/block_based_table_reader.cc#L1369 > > Iterator *Seek*: it will do binary search, by default we don't specify > prefix feature(https://github.com/facebook/rocksdb/wiki/Prefix-Seek-API-Changes). > https://github.com/facebook/rocksdb/blob/master/table/block.cc#L94 > > So I use a simple tests: > > ./db_bench -num 10000000 -benchmarks fillbatch > fill the db firstly with 1000w records. > > ./db_bench -use_existing_db -benchmarks readrandomfast > readrandomfast case will use *Get* API to retrive data > > [root@hunter-node2 rocksdb]# ./db_bench -use_existing_db -benchmarks > readrandomfast > > LevelDB: version 4.3 > Date: Wed Jun 1 00:29:16 2016 > CPU: 32 * Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz > CPUCache: 20480 KB > Keys: 16 bytes each > Values: 100 bytes each (50 bytes after compression) > Entries: 1000000 > Prefix: 0 bytes > Keys per prefix: 0 > RawSize: 110.6 MB (estimated) > FileSize: 62.9 MB (estimated) > Writes per second: 0 > Compression: Snappy > Memtablerep: skip_list > Perf Level: 0 > WARNING: Assertions are enabled; benchmarks unnecessarily slow > ------------------------------------------------ > DB path: [/tmp/rocksdbtest-0/dbbench] > readrandomfast : 4.570 micros/op 218806 ops/sec; (1000100 of > 1000100 found, issued 46639 non-exist keys) > > =========================== > then I modify readrandomfast to use Iterator API[0]: > > [root@hunter-node2 rocksdb]# ./db_bench -use_existing_db -benchmarks > readrandomfast > LevelDB: version 4.3 > Date: Wed Jun 1 00:33:03 2016 > CPU: 32 * Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz > CPUCache: 20480 KB > Keys: 16 bytes each > Values: 100 bytes each (50 bytes after compression) > Entries: 1000000 > Prefix: 0 bytes > Keys per prefix: 0 > RawSize: 110.6 MB (estimated) > FileSize: 62.9 MB (estimated) > Writes per second: 0 > Compression: Snappy > Memtablerep: skip_list > Perf Level: 0 > WARNING: Assertions are enabled; benchmarks unnecessarily slow > ------------------------------------------------ > DB path: [/tmp/rocksdbtest-0/dbbench] > readrandomfast : 45.188 micros/op 22129 ops/sec; (1000100 of > 1000100 found, issued 46639 non-exist keys) > > > 45.18 us/op vs 4.57us/op! > > The test can be repeated and easy to do! Plz correct if I'm doing > foolish thing I'm not aware.. Excellent catch Haomai! I'm not sure I will be able to test before I leave on holiday, but if I do I will report back. Do you think upstream rocksdb can be improved to make the iterator implementation faster? > > So I proposal this PR: https://github.com/ceph/ceph/pull/9411 > > We still can make further improvements by scanning all iterate usage > to make it better! > > [0]: > --- a/db/db_bench.cc > +++ b/db/db_bench.cc > @@ -2923,14 +2923,12 @@ class Benchmark { > int64_t key_rand = thread->rand.Next() & (pot - 1); > GenerateKeyFromInt(key_rand, FLAGS_num, &key); > ++read; > - auto status = db->Get(options, key, &value); > - if (status.ok()) { > - ++found; > - } else if (!status.IsNotFound()) { > - fprintf(stderr, "Get returned an error: %s\n", > - status.ToString().c_str()); > - abort(); > - } > + Iterator* iter = db->NewIterator(options); > + iter->Seek(key); > + if (iter->Valid() && iter->key().compare(key) == 0) { > + found++; > + } > + > if (key_rand >= FLAGS_num) { > ++nonexist; > } > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Jun 01, 2016 at 12:49:53AM +0800, Haomai Wang wrote: > Hi Sage and Mark, > > As mentioned in BlueStore standup, I found rocksdb iterator *Seek* > won't use bloom filter like *Get*. > > *Get* impl: it will look at filter firstly > https://github.com/facebook/rocksdb/blob/master/table/block_based_table_reader.cc#L1369 > > Iterator *Seek*: it will do binary search, by default we don't specify > prefix feature(https://github.com/facebook/rocksdb/wiki/Prefix-Seek-API-Changes). > https://github.com/facebook/rocksdb/blob/master/table/block.cc#L94 > > [..] > --- a/db/db_bench.cc > +++ b/db/db_bench.cc > @@ -2923,14 +2923,12 @@ class Benchmark { > int64_t key_rand = thread->rand.Next() & (pot - 1); > GenerateKeyFromInt(key_rand, FLAGS_num, &key); > ++read; > - auto status = db->Get(options, key, &value); > - if (status.ok()) { > - ++found; > - } else if (!status.IsNotFound()) { > - fprintf(stderr, "Get returned an error: %s\n", > - status.ToString().c_str()); > - abort(); > - } > + Iterator* iter = db->NewIterator(options); > + iter->Seek(key); > + if (iter->Valid() && iter->key().compare(key) == 0) { > + found++; > + } > + > if (key_rand >= FLAGS_num) { > ++nonexist; > } Aren't you missing "delete iter" here?
On Wed, Jun 1, 2016 at 1:17 AM, Piotr Dałek <branch@predictor.org.pl> wrote: > On Wed, Jun 01, 2016 at 12:49:53AM +0800, Haomai Wang wrote: >> Hi Sage and Mark, >> >> As mentioned in BlueStore standup, I found rocksdb iterator *Seek* >> won't use bloom filter like *Get*. >> >> *Get* impl: it will look at filter firstly >> https://github.com/facebook/rocksdb/blob/master/table/block_based_table_reader.cc#L1369 >> >> Iterator *Seek*: it will do binary search, by default we don't specify >> prefix feature(https://github.com/facebook/rocksdb/wiki/Prefix-Seek-API-Changes). >> https://github.com/facebook/rocksdb/blob/master/table/block.cc#L94 >> >> [..] >> --- a/db/db_bench.cc >> +++ b/db/db_bench.cc >> @@ -2923,14 +2923,12 @@ class Benchmark { >> int64_t key_rand = thread->rand.Next() & (pot - 1); >> GenerateKeyFromInt(key_rand, FLAGS_num, &key); >> ++read; >> - auto status = db->Get(options, key, &value); >> - if (status.ok()) { >> - ++found; >> - } else if (!status.IsNotFound()) { >> - fprintf(stderr, "Get returned an error: %s\n", >> - status.ToString().c_str()); >> - abort(); >> - } >> + Iterator* iter = db->NewIterator(options); >> + iter->Seek(key); >> + if (iter->Valid() && iter->key().compare(key) == 0) { >> + found++; >> + } >> + >> if (key_rand >= FLAGS_num) { >> ++nonexist; >> } > > Aren't you missing "delete iter" here? oh, sure. I retest this, now it's 20us/op. Much better :-). 20 vs 4. And actually this test set is seq int key and the value is small, it will be make the gap not so much. Consider the binary search vs bloom filter, I think the latency is huge. We shouldn't use any iterate except we ensure it's a continuous key set. > > -- > Piotr Dałek > branch@predictor.org.pl > http://blog.predictor.org.pl > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Haomai, I noticed this as well, and made same changes to RocksDBStore in this PR last week: https://github.com/ceph/ceph/pull/9215 One thing which is even worse, seek will bypass row cache, so kv pairs won't be able to be cached in row cache. I am working to benchmark the performance impact, will publish the results after I am done this week. Jianjian On Tue, May 31, 2016 at 9:49 AM, Haomai Wang <haomai@xsky.com> wrote: > Hi Sage and Mark, > > As mentioned in BlueStore standup, I found rocksdb iterator *Seek* > won't use bloom filter like *Get*. > > *Get* impl: it will look at filter firstly > https://github.com/facebook/rocksdb/blob/master/table/block_based_table_reader.cc#L1369 > > Iterator *Seek*: it will do binary search, by default we don't specify > prefix feature(https://github.com/facebook/rocksdb/wiki/Prefix-Seek-API-Changes). > https://github.com/facebook/rocksdb/blob/master/table/block.cc#L94 > > So I use a simple tests: > > ./db_bench -num 10000000 -benchmarks fillbatch > fill the db firstly with 1000w records. > > ./db_bench -use_existing_db -benchmarks readrandomfast > readrandomfast case will use *Get* API to retrive data > > [root@hunter-node2 rocksdb]# ./db_bench -use_existing_db -benchmarks > readrandomfast > > LevelDB: version 4.3 > Date: Wed Jun 1 00:29:16 2016 > CPU: 32 * Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz > CPUCache: 20480 KB > Keys: 16 bytes each > Values: 100 bytes each (50 bytes after compression) > Entries: 1000000 > Prefix: 0 bytes > Keys per prefix: 0 > RawSize: 110.6 MB (estimated) > FileSize: 62.9 MB (estimated) > Writes per second: 0 > Compression: Snappy > Memtablerep: skip_list > Perf Level: 0 > WARNING: Assertions are enabled; benchmarks unnecessarily slow > ------------------------------------------------ > DB path: [/tmp/rocksdbtest-0/dbbench] > readrandomfast : 4.570 micros/op 218806 ops/sec; (1000100 of > 1000100 found, issued 46639 non-exist keys) > > =========================== > then I modify readrandomfast to use Iterator API[0]: > > [root@hunter-node2 rocksdb]# ./db_bench -use_existing_db -benchmarks > readrandomfast > LevelDB: version 4.3 > Date: Wed Jun 1 00:33:03 2016 > CPU: 32 * Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz > CPUCache: 20480 KB > Keys: 16 bytes each > Values: 100 bytes each (50 bytes after compression) > Entries: 1000000 > Prefix: 0 bytes > Keys per prefix: 0 > RawSize: 110.6 MB (estimated) > FileSize: 62.9 MB (estimated) > Writes per second: 0 > Compression: Snappy > Memtablerep: skip_list > Perf Level: 0 > WARNING: Assertions are enabled; benchmarks unnecessarily slow > ------------------------------------------------ > DB path: [/tmp/rocksdbtest-0/dbbench] > readrandomfast : 45.188 micros/op 22129 ops/sec; (1000100 of > 1000100 found, issued 46639 non-exist keys) > > > 45.18 us/op vs 4.57us/op! > > The test can be repeated and easy to do! Plz correct if I'm doing > foolish thing I'm not aware.. > > So I proposal this PR: https://github.com/ceph/ceph/pull/9411 > > We still can make further improvements by scanning all iterate usage > to make it better! > > [0]: > --- a/db/db_bench.cc > +++ b/db/db_bench.cc > @@ -2923,14 +2923,12 @@ class Benchmark { > int64_t key_rand = thread->rand.Next() & (pot - 1); > GenerateKeyFromInt(key_rand, FLAGS_num, &key); > ++read; > - auto status = db->Get(options, key, &value); > - if (status.ok()) { > - ++found; > - } else if (!status.IsNotFound()) { > - fprintf(stderr, "Get returned an error: %s\n", > - status.ToString().c_str()); > - abort(); > - } > + Iterator* iter = db->NewIterator(options); > + iter->Seek(key); > + if (iter->Valid() && iter->key().compare(key) == 0) { > + found++; > + } > + > if (key_rand >= FLAGS_num) { > ++nonexist; > } > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Jun 1, 2016 at 2:08 AM, Jianjian Huo <jianjian.huo@samsung.com> wrote: > Hi Haomai, > > I noticed this as well, and made same changes to RocksDBStore in this PR last week: > https://github.com/ceph/ceph/pull/9215 > > One thing which is even worse, seek will bypass row cache, so kv pairs won't be able to be cached in row cache. > I am working to benchmark the performance impact, will publish the results after I am done this week. Oh, cool! I think you can cherry-pick my another leveldb fix. BTW, do you pay an attention to prefix seek api? I think it will be more suitable than column family in ceph case. If we can have well-defined prefix rule, we can make most of range query cheaper! > > Jianjian > > On Tue, May 31, 2016 at 9:49 AM, Haomai Wang <haomai@xsky.com> wrote: >> Hi Sage and Mark, >> >> As mentioned in BlueStore standup, I found rocksdb iterator *Seek* >> won't use bloom filter like *Get*. >> >> *Get* impl: it will look at filter firstly >> https://github.com/facebook/rocksdb/blob/master/table/block_based_table_reader.cc#L1369 >> >> Iterator *Seek*: it will do binary search, by default we don't specify >> prefix feature(https://github.com/facebook/rocksdb/wiki/Prefix-Seek-API-Changes). >> https://github.com/facebook/rocksdb/blob/master/table/block.cc#L94 >> >> So I use a simple tests: >> >> ./db_bench -num 10000000 -benchmarks fillbatch >> fill the db firstly with 1000w records. >> >> ./db_bench -use_existing_db -benchmarks readrandomfast >> readrandomfast case will use *Get* API to retrive data >> >> [root@hunter-node2 rocksdb]# ./db_bench -use_existing_db -benchmarks >> readrandomfast >> >> LevelDB: version 4.3 >> Date: Wed Jun 1 00:29:16 2016 >> CPU: 32 * Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz >> CPUCache: 20480 KB >> Keys: 16 bytes each >> Values: 100 bytes each (50 bytes after compression) >> Entries: 1000000 >> Prefix: 0 bytes >> Keys per prefix: 0 >> RawSize: 110.6 MB (estimated) >> FileSize: 62.9 MB (estimated) >> Writes per second: 0 >> Compression: Snappy >> Memtablerep: skip_list >> Perf Level: 0 >> WARNING: Assertions are enabled; benchmarks unnecessarily slow >> ------------------------------------------------ >> DB path: [/tmp/rocksdbtest-0/dbbench] >> readrandomfast : 4.570 micros/op 218806 ops/sec; (1000100 of >> 1000100 found, issued 46639 non-exist keys) >> >> =========================== >> then I modify readrandomfast to use Iterator API[0]: >> >> [root@hunter-node2 rocksdb]# ./db_bench -use_existing_db -benchmarks >> readrandomfast >> LevelDB: version 4.3 >> Date: Wed Jun 1 00:33:03 2016 >> CPU: 32 * Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz >> CPUCache: 20480 KB >> Keys: 16 bytes each >> Values: 100 bytes each (50 bytes after compression) >> Entries: 1000000 >> Prefix: 0 bytes >> Keys per prefix: 0 >> RawSize: 110.6 MB (estimated) >> FileSize: 62.9 MB (estimated) >> Writes per second: 0 >> Compression: Snappy >> Memtablerep: skip_list >> Perf Level: 0 >> WARNING: Assertions are enabled; benchmarks unnecessarily slow >> ------------------------------------------------ >> DB path: [/tmp/rocksdbtest-0/dbbench] >> readrandomfast : 45.188 micros/op 22129 ops/sec; (1000100 of >> 1000100 found, issued 46639 non-exist keys) >> >> >> 45.18 us/op vs 4.57us/op! >> >> The test can be repeated and easy to do! Plz correct if I'm doing >> foolish thing I'm not aware.. >> >> So I proposal this PR: https://github.com/ceph/ceph/pull/9411 >> >> We still can make further improvements by scanning all iterate usage >> to make it better! >> >> [0]: >> --- a/db/db_bench.cc >> +++ b/db/db_bench.cc >> @@ -2923,14 +2923,12 @@ class Benchmark { >> int64_t key_rand = thread->rand.Next() & (pot - 1); >> GenerateKeyFromInt(key_rand, FLAGS_num, &key); >> ++read; >> - auto status = db->Get(options, key, &value); >> - if (status.ok()) { >> - ++found; >> - } else if (!status.IsNotFound()) { >> - fprintf(stderr, "Get returned an error: %s\n", >> - status.ToString().c_str()); >> - abort(); >> - } >> + Iterator* iter = db->NewIterator(options); >> + iter->Seek(key); >> + if (iter->Valid() && iter->key().compare(key) == 0) { >> + found++; >> + } >> + >> if (key_rand >= FLAGS_num) { >> ++nonexist; >> } >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, May 31, 2016 at 11:12 AM, Haomai Wang <haomai@xsky.com> wrote: > On Wed, Jun 1, 2016 at 2:08 AM, Jianjian Huo <jianjian.huo@samsung.com> wrote: >> Hi Haomai, >> >> I noticed this as well, and made same changes to RocksDBStore in this PR last week: >> https://github.com/ceph/ceph/pull/9215 >> >> One thing which is even worse, seek will bypass row cache, so kv pairs won't be able to be cached in row cache. >> I am working to benchmark the performance impact, will publish the results after I am done this week. > > Oh, cool! I think you can cherry-pick my another leveldb fix. No problem, I will do that. > > BTW, do you pay an attention to prefix seek api? I think it will be > more suitable than column family in ceph case. If we can have > well-defined prefix rule, we can make most of range query cheaper! When keys with same prefix are stored in their own SST files(CF case), even seeking without prefix will be faster than seeking with prefix but mixed with different prefixed keys? I am not sure what optimization prefix seek will use internally for block based format, but to me, it's hard to beat the case when you only have one prefixed keys stored separately. > >> >> Jianjian >> >> On Tue, May 31, 2016 at 9:49 AM, Haomai Wang <haomai@xsky.com> wrote: >>> Hi Sage and Mark, >>> >>> As mentioned in BlueStore standup, I found rocksdb iterator *Seek* >>> won't use bloom filter like *Get*. >>> >>> *Get* impl: it will look at filter firstly >>> https://github.com/facebook/rocksdb/blob/master/table/block_based_table_reader.cc#L1369 >>> >>> Iterator *Seek*: it will do binary search, by default we don't specify >>> prefix feature(https://github.com/facebook/rocksdb/wiki/Prefix-Seek-API-Changes). >>> https://github.com/facebook/rocksdb/blob/master/table/block.cc#L94 >>> >>> So I use a simple tests: >>> >>> ./db_bench -num 10000000 -benchmarks fillbatch >>> fill the db firstly with 1000w records. >>> >>> ./db_bench -use_existing_db -benchmarks readrandomfast >>> readrandomfast case will use *Get* API to retrive data >>> >>> [root@hunter-node2 rocksdb]# ./db_bench -use_existing_db -benchmarks >>> readrandomfast >>> >>> LevelDB: version 4.3 >>> Date: Wed Jun 1 00:29:16 2016 >>> CPU: 32 * Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz >>> CPUCache: 20480 KB >>> Keys: 16 bytes each >>> Values: 100 bytes each (50 bytes after compression) >>> Entries: 1000000 >>> Prefix: 0 bytes >>> Keys per prefix: 0 >>> RawSize: 110.6 MB (estimated) >>> FileSize: 62.9 MB (estimated) >>> Writes per second: 0 >>> Compression: Snappy >>> Memtablerep: skip_list >>> Perf Level: 0 >>> WARNING: Assertions are enabled; benchmarks unnecessarily slow >>> ------------------------------------------------ >>> DB path: [/tmp/rocksdbtest-0/dbbench] >>> readrandomfast : 4.570 micros/op 218806 ops/sec; (1000100 of >>> 1000100 found, issued 46639 non-exist keys) >>> >>> =========================== >>> then I modify readrandomfast to use Iterator API[0]: >>> >>> [root@hunter-node2 rocksdb]# ./db_bench -use_existing_db -benchmarks >>> readrandomfast >>> LevelDB: version 4.3 >>> Date: Wed Jun 1 00:33:03 2016 >>> CPU: 32 * Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz >>> CPUCache: 20480 KB >>> Keys: 16 bytes each >>> Values: 100 bytes each (50 bytes after compression) >>> Entries: 1000000 >>> Prefix: 0 bytes >>> Keys per prefix: 0 >>> RawSize: 110.6 MB (estimated) >>> FileSize: 62.9 MB (estimated) >>> Writes per second: 0 >>> Compression: Snappy >>> Memtablerep: skip_list >>> Perf Level: 0 >>> WARNING: Assertions are enabled; benchmarks unnecessarily slow >>> ------------------------------------------------ >>> DB path: [/tmp/rocksdbtest-0/dbbench] >>> readrandomfast : 45.188 micros/op 22129 ops/sec; (1000100 of >>> 1000100 found, issued 46639 non-exist keys) >>> >>> >>> 45.18 us/op vs 4.57us/op! >>> >>> The test can be repeated and easy to do! Plz correct if I'm doing >>> foolish thing I'm not aware.. >>> >>> So I proposal this PR: https://github.com/ceph/ceph/pull/9411 >>> >>> We still can make further improvements by scanning all iterate usage >>> to make it better! >>> >>> [0]: >>> --- a/db/db_bench.cc >>> +++ b/db/db_bench.cc >>> @@ -2923,14 +2923,12 @@ class Benchmark { >>> int64_t key_rand = thread->rand.Next() & (pot - 1); >>> GenerateKeyFromInt(key_rand, FLAGS_num, &key); >>> ++read; >>> - auto status = db->Get(options, key, &value); >>> - if (status.ok()) { >>> - ++found; >>> - } else if (!status.IsNotFound()) { >>> - fprintf(stderr, "Get returned an error: %s\n", >>> - status.ToString().c_str()); >>> - abort(); >>> - } >>> + Iterator* iter = db->NewIterator(options); >>> + iter->Seek(key); >>> + if (iter->Valid() && iter->key().compare(key) == 0) { >>> + found++; >>> + } >>> + >>> if (key_rand >= FLAGS_num) { >>> ++nonexist; >>> } >>> -- >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Jun 1, 2016 at 3:40 AM, Jianjian Huo <jianjian.huo@samsung.com> wrote: > > On Tue, May 31, 2016 at 11:12 AM, Haomai Wang <haomai@xsky.com> wrote: >> On Wed, Jun 1, 2016 at 2:08 AM, Jianjian Huo <jianjian.huo@samsung.com> wrote: >>> Hi Haomai, >>> >>> I noticed this as well, and made same changes to RocksDBStore in this PR last week: >>> https://github.com/ceph/ceph/pull/9215 >>> >>> One thing which is even worse, seek will bypass row cache, so kv pairs won't be able to be cached in row cache. >>> I am working to benchmark the performance impact, will publish the results after I am done this week. >> >> Oh, cool! I think you can cherry-pick my another leveldb fix. > > No problem, I will do that. >> >> BTW, do you pay an attention to prefix seek api? I think it will be >> more suitable than column family in ceph case. If we can have >> well-defined prefix rule, we can make most of range query cheaper! > > When keys with same prefix are stored in their own SST files(CF case), even seeking without prefix will be faster than seeking with prefix but mixed with different prefixed keys? From my current view, prefix could be more flexible. For example, each rgw bucket index could use one prefix to make each bucket index object seek separated. For CF, it would be too heavry. > I am not sure what optimization prefix seek will use internally for block based format, but to me, it's hard to beat the case when you only have one prefixed keys stored separately. >> >>> >>> Jianjian >>> >>> On Tue, May 31, 2016 at 9:49 AM, Haomai Wang <haomai@xsky.com> wrote: >>>> Hi Sage and Mark, >>>> >>>> As mentioned in BlueStore standup, I found rocksdb iterator *Seek* >>>> won't use bloom filter like *Get*. >>>> >>>> *Get* impl: it will look at filter firstly >>>> https://github.com/facebook/rocksdb/blob/master/table/block_based_table_reader.cc#L1369 >>>> >>>> Iterator *Seek*: it will do binary search, by default we don't specify >>>> prefix feature(https://github.com/facebook/rocksdb/wiki/Prefix-Seek-API-Changes). >>>> https://github.com/facebook/rocksdb/blob/master/table/block.cc#L94 >>>> >>>> So I use a simple tests: >>>> >>>> ./db_bench -num 10000000 -benchmarks fillbatch >>>> fill the db firstly with 1000w records. >>>> >>>> ./db_bench -use_existing_db -benchmarks readrandomfast >>>> readrandomfast case will use *Get* API to retrive data >>>> >>>> [root@hunter-node2 rocksdb]# ./db_bench -use_existing_db -benchmarks >>>> readrandomfast >>>> >>>> LevelDB: version 4.3 >>>> Date: Wed Jun 1 00:29:16 2016 >>>> CPU: 32 * Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz >>>> CPUCache: 20480 KB >>>> Keys: 16 bytes each >>>> Values: 100 bytes each (50 bytes after compression) >>>> Entries: 1000000 >>>> Prefix: 0 bytes >>>> Keys per prefix: 0 >>>> RawSize: 110.6 MB (estimated) >>>> FileSize: 62.9 MB (estimated) >>>> Writes per second: 0 >>>> Compression: Snappy >>>> Memtablerep: skip_list >>>> Perf Level: 0 >>>> WARNING: Assertions are enabled; benchmarks unnecessarily slow >>>> ------------------------------------------------ >>>> DB path: [/tmp/rocksdbtest-0/dbbench] >>>> readrandomfast : 4.570 micros/op 218806 ops/sec; (1000100 of >>>> 1000100 found, issued 46639 non-exist keys) >>>> >>>> =========================== >>>> then I modify readrandomfast to use Iterator API[0]: >>>> >>>> [root@hunter-node2 rocksdb]# ./db_bench -use_existing_db -benchmarks >>>> readrandomfast >>>> LevelDB: version 4.3 >>>> Date: Wed Jun 1 00:33:03 2016 >>>> CPU: 32 * Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz >>>> CPUCache: 20480 KB >>>> Keys: 16 bytes each >>>> Values: 100 bytes each (50 bytes after compression) >>>> Entries: 1000000 >>>> Prefix: 0 bytes >>>> Keys per prefix: 0 >>>> RawSize: 110.6 MB (estimated) >>>> FileSize: 62.9 MB (estimated) >>>> Writes per second: 0 >>>> Compression: Snappy >>>> Memtablerep: skip_list >>>> Perf Level: 0 >>>> WARNING: Assertions are enabled; benchmarks unnecessarily slow >>>> ------------------------------------------------ >>>> DB path: [/tmp/rocksdbtest-0/dbbench] >>>> readrandomfast : 45.188 micros/op 22129 ops/sec; (1000100 of >>>> 1000100 found, issued 46639 non-exist keys) >>>> >>>> >>>> 45.18 us/op vs 4.57us/op! >>>> >>>> The test can be repeated and easy to do! Plz correct if I'm doing >>>> foolish thing I'm not aware.. >>>> >>>> So I proposal this PR: https://github.com/ceph/ceph/pull/9411 >>>> >>>> We still can make further improvements by scanning all iterate usage >>>> to make it better! >>>> >>>> [0]: >>>> --- a/db/db_bench.cc >>>> +++ b/db/db_bench.cc >>>> @@ -2923,14 +2923,12 @@ class Benchmark { >>>> int64_t key_rand = thread->rand.Next() & (pot - 1); >>>> GenerateKeyFromInt(key_rand, FLAGS_num, &key); >>>> ++read; >>>> - auto status = db->Get(options, key, &value); >>>> - if (status.ok()) { >>>> - ++found; >>>> - } else if (!status.IsNotFound()) { >>>> - fprintf(stderr, "Get returned an error: %s\n", >>>> - status.ToString().c_str()); >>>> - abort(); >>>> - } >>>> + Iterator* iter = db->NewIterator(options); >>>> + iter->Seek(key); >>>> + if (iter->Valid() && iter->key().compare(key) == 0) { >>>> + found++; >>>> + } >>>> + >>>> if (key_rand >= FLAGS_num) { >>>> ++nonexist; >>>> } >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>>> the body of a message to majordomo@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
PiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBjZXBoLWRldmVsLW93bmVyQHZn ZXIua2VybmVsLm9yZyBbbWFpbHRvOmNlcGgtZGV2ZWwtDQo+IG93bmVyQHZnZXIua2VybmVsLm9y Z10gT24gQmVoYWxmIE9mIEhhb21haSBXYW5nDQo+IFNlbnQ6IFR1ZXNkYXksIE1heSAzMSwgMjAx NiA3OjM4IFBNDQo+IFRvOiBKaWFuamlhbiBIdW8gPGppYW5qaWFuLmh1b0BzYW1zdW5nLmNvbT4N Cj4gQ2M6IFNhZ2UgV2VpbCA8c3dlaWxAcmVkaGF0LmNvbT47IE1hcmsgTmVsc29uIDxtbmVsc29u QHJlZGhhdC5jb20+Ow0KPiBjZXBoLWRldmVsQHZnZXIua2VybmVsLm9yZw0KPiBTdWJqZWN0OiBS ZTogUm9ja3NEQiBJbmNvcnJlY3QgQVBJIFVzYWdlDQo+IA0KPiBPbiBXZWQsIEp1biAxLCAyMDE2 IGF0IDM6NDAgQU0sIEppYW5qaWFuIEh1byA8amlhbmppYW4uaHVvQHNhbXN1bmcuY29tPg0KPiB3 cm90ZToNCj4gPg0KPiA+IE9uIFR1ZSwgTWF5IDMxLCAyMDE2IGF0IDExOjEyIEFNLCBIYW9tYWkg V2FuZyA8aGFvbWFpQHhza3kuY29tPg0KPiB3cm90ZToNCj4gPj4gT24gV2VkLCBKdW4gMSwgMjAx NiBhdCAyOjA4IEFNLCBKaWFuamlhbiBIdW8NCj4gPGppYW5qaWFuLmh1b0BzYW1zdW5nLmNvbT4g d3JvdGU6DQo+ID4+PiBIaSBIYW9tYWksDQo+ID4+Pg0KPiA+Pj4gSSBub3RpY2VkIHRoaXMgYXMg d2VsbCwgYW5kIG1hZGUgc2FtZSBjaGFuZ2VzIHRvIFJvY2tzREJTdG9yZSBpbiB0aGlzIFBSDQo+ IGxhc3Qgd2VlazoNCj4gPj4+IGh0dHBzOi8vZ2l0aHViLmNvbS9jZXBoL2NlcGgvcHVsbC85MjE1 DQo+ID4+Pg0KPiA+Pj4gT25lIHRoaW5nIHdoaWNoIGlzIGV2ZW4gd29yc2UsICBzZWVrIHdpbGwg YnlwYXNzIHJvdyBjYWNoZSwgc28ga3YgcGFpcnMNCj4gd29uJ3QgYmUgYWJsZSB0byBiZSBjYWNo ZWQgaW4gcm93IGNhY2hlLg0KPiA+Pj4gSSBhbSB3b3JraW5nIHRvIGJlbmNobWFyayB0aGUgcGVy Zm9ybWFuY2UgaW1wYWN0LCB3aWxsIHB1Ymxpc2ggdGhlDQo+IHJlc3VsdHMgYWZ0ZXIgSSBhbSBk b25lIHRoaXMgd2Vlay4NCj4gPj4NCj4gPj4gT2gsIGNvb2whIEkgdGhpbmsgeW91IGNhbiBjaGVy cnktcGljayBteSBhbm90aGVyIGxldmVsZGIgZml4Lg0KPiA+DQo+ID4gTm8gcHJvYmxlbSwgSSB3 aWxsIGRvIHRoYXQuDQo+ID4+DQo+ID4+IEJUVywgZG8geW91IHBheSBhbiBhdHRlbnRpb24gdG8g cHJlZml4IHNlZWsgYXBpPyBJIHRoaW5rIGl0IHdpbGwgYmUNCj4gPj4gbW9yZSBzdWl0YWJsZSB0 aGFuIGNvbHVtbiBmYW1pbHkgaW4gY2VwaCBjYXNlLiBJZiB3ZSBjYW4gaGF2ZQ0KPiA+PiB3ZWxs LWRlZmluZWQgcHJlZml4IHJ1bGUsIHdlIGNhbiBtYWtlIG1vc3Qgb2YgcmFuZ2UgcXVlcnkgY2hl YXBlciENCj4gPg0KPiA+IFdoZW4ga2V5cyB3aXRoIHNhbWUgcHJlZml4IGFyZSBzdG9yZWQgaW4g dGhlaXIgb3duIFNTVCBmaWxlcyhDRiBjYXNlKSwNCj4gZXZlbiBzZWVraW5nIHdpdGhvdXQgcHJl Zml4IHdpbGwgYmUgZmFzdGVyIHRoYW4gc2Vla2luZyB3aXRoIHByZWZpeCBidXQgbWl4ZWQNCj4g d2l0aCBkaWZmZXJlbnQgcHJlZml4ZWQga2V5cz8NCj4gDQo+IEZyb20gbXkgY3VycmVudCB2aWV3 LCBwcmVmaXggY291bGQgYmUgbW9yZSBmbGV4aWJsZS4gRm9yIGV4YW1wbGUsIGVhY2ggcmd3DQo+ IGJ1Y2tldCBpbmRleCBjb3VsZCB1c2Ugb25lIHByZWZpeCB0byBtYWtlIGVhY2ggYnVja2V0IGlu ZGV4IG9iamVjdCBzZWVrDQo+IHNlcGFyYXRlZC4gRm9yIENGLCBpdCB3b3VsZCBiZSB0b28gaGVh dnJ5Lg0KDQpUaGlzIG1pZ2h0IGNhdXNlIG90aGVyIHByb2JsZW1zLiBXb3VsZCBpdCBkcmFtYXRp Y2FsbHkgaW5jcmVhc2UgdGhlIG51bWJlciBvZiBmaWxlcyB0aGF0IEJsdWVGUyBuZWVkcyB0byBt YW5hZ2U/IElmIHNvLCB0aGF0IG1pZ2h0IGVmZmVjdGl2ZWx5IGJyZWFrIHRoYXQgY29kZSB0b28g KG9mIGNvdXJzZSBpdCdzIGZpeGFibGUgYWxzbyA6KSkNCj4gDQo+ID4gSSBhbSBub3Qgc3VyZSB3 aGF0IG9wdGltaXphdGlvbiBwcmVmaXggc2VlayB3aWxsIHVzZSBpbnRlcm5hbGx5IGZvciBibG9j aw0KPiBiYXNlZCBmb3JtYXQsIGJ1dCB0byBtZSwgaXQncyBoYXJkIHRvIGJlYXQgdGhlIGNhc2Ug d2hlbiB5b3Ugb25seSBoYXZlIG9uZQ0KPiBwcmVmaXhlZCBrZXlzIHN0b3JlZCBzZXBhcmF0ZWx5 Lg0KPiA+Pg0KPiA+Pj4NCj4gPj4+IEppYW5qaWFuDQo+ID4+Pg0KPiA+Pj4gT24gVHVlLCBNYXkg MzEsIDIwMTYgYXQgOTo0OSBBTSwgSGFvbWFpIFdhbmcgPGhhb21haUB4c2t5LmNvbT4NCj4gd3Jv dGU6DQo+ID4+Pj4gSGkgU2FnZSBhbmQgTWFyaywNCj4gPj4+Pg0KPiA+Pj4+IEFzIG1lbnRpb25l ZCBpbiBCbHVlU3RvcmUgc3RhbmR1cCwgSSBmb3VuZCByb2Nrc2RiIGl0ZXJhdG9yICpTZWVrKg0K PiA+Pj4+IHdvbid0IHVzZSBibG9vbSBmaWx0ZXIgbGlrZSAqR2V0Ki4NCj4gPj4+Pg0KPiA+Pj4+ ICpHZXQqIGltcGw6IGl0IHdpbGwgbG9vayBhdCBmaWx0ZXIgZmlyc3RseQ0KPiA+Pj4+DQo+IGh0 dHBzOi8vZ2l0aHViLmNvbS9mYWNlYm9vay9yb2Nrc2RiL2Jsb2IvbWFzdGVyL3RhYmxlL2Jsb2Nr X2Jhc2VkX3QNCj4gPj4+PiBhYmxlX3JlYWRlci5jYyNMMTM2OQ0KPiA+Pj4+DQo+ID4+Pj4gSXRl cmF0b3IgKlNlZWsqOiBpdCB3aWxsIGRvIGJpbmFyeSBzZWFyY2gsIGJ5IGRlZmF1bHQgd2UgZG9u J3QNCj4gPj4+PiBzcGVjaWZ5IHByZWZpeA0KPiBmZWF0dXJlKGh0dHBzOi8vZ2l0aHViLmNvbS9m YWNlYm9vay9yb2Nrc2RiL3dpa2kvUHJlZml4LVNlZWstQVBJLQ0KPiBDaGFuZ2VzKS4NCj4gPj4+ PiBodHRwczovL2dpdGh1Yi5jb20vZmFjZWJvb2svcm9ja3NkYi9ibG9iL21hc3Rlci90YWJsZS9i bG9jay5jYyNMOTQNCj4gPj4+Pg0KPiA+Pj4+IFNvIEkgdXNlIGEgc2ltcGxlIHRlc3RzOg0KPiA+ Pj4+DQo+ID4+Pj4gLi9kYl9iZW5jaCAtbnVtIDEwMDAwMDAwICAtYmVuY2htYXJrcyBmaWxsYmF0 Y2ggZmlsbCB0aGUgZGIgZmlyc3RseQ0KPiA+Pj4+IHdpdGggMTAwMHcgcmVjb3Jkcy4NCj4gPj4+ Pg0KPiA+Pj4+IC4vZGJfYmVuY2ggLXVzZV9leGlzdGluZ19kYiAgLWJlbmNobWFya3MgcmVhZHJh bmRvbWZhc3QNCj4gPj4+PiByZWFkcmFuZG9tZmFzdCBjYXNlIHdpbGwgdXNlICpHZXQqIEFQSSB0 byByZXRyaXZlIGRhdGENCj4gPj4+Pg0KPiA+Pj4+IFtyb290QGh1bnRlci1ub2RlMiByb2Nrc2Ri XSMgLi9kYl9iZW5jaCAtdXNlX2V4aXN0aW5nX2RiDQo+ID4+Pj4gLWJlbmNobWFya3MgcmVhZHJh bmRvbWZhc3QNCj4gPj4+Pg0KPiA+Pj4+IExldmVsREI6ICAgIHZlcnNpb24gNC4zDQo+ID4+Pj4g RGF0ZTogICAgICAgV2VkIEp1biAgMSAwMDoyOToxNiAyMDE2DQo+ID4+Pj4gQ1BVOiAgICAgICAg MzIgKiBJbnRlbChSKSBYZW9uKFIpIENQVSBFNS0yNjgwIDAgQCAyLjcwR0h6DQo+ID4+Pj4gQ1BV Q2FjaGU6ICAgMjA0ODAgS0INCj4gPj4+PiBLZXlzOiAgICAgICAxNiBieXRlcyBlYWNoDQo+ID4+ Pj4gVmFsdWVzOiAgICAgMTAwIGJ5dGVzIGVhY2ggKDUwIGJ5dGVzIGFmdGVyIGNvbXByZXNzaW9u KQ0KPiA+Pj4+IEVudHJpZXM6ICAgIDEwMDAwMDANCj4gPj4+PiBQcmVmaXg6ICAgIDAgYnl0ZXMN Cj4gPj4+PiBLZXlzIHBlciBwcmVmaXg6ICAgIDANCj4gPj4+PiBSYXdTaXplOiAgICAxMTAuNiBN QiAoZXN0aW1hdGVkKQ0KPiA+Pj4+IEZpbGVTaXplOiAgIDYyLjkgTUIgKGVzdGltYXRlZCkNCj4g Pj4+PiBXcml0ZXMgcGVyIHNlY29uZDogMA0KPiA+Pj4+IENvbXByZXNzaW9uOiBTbmFwcHkNCj4g Pj4+PiBNZW10YWJsZXJlcDogc2tpcF9saXN0DQo+ID4+Pj4gUGVyZiBMZXZlbDogMA0KPiA+Pj4+ IFdBUk5JTkc6IEFzc2VydGlvbnMgYXJlIGVuYWJsZWQ7IGJlbmNobWFya3MgdW5uZWNlc3Nhcmls eSBzbG93DQo+ID4+Pj4gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tDQo+ID4+Pj4gREIgcGF0aDogWy90bXAvcm9ja3NkYnRlc3QtMC9kYmJlbmNoXQ0KPiA+ Pj4+IHJlYWRyYW5kb21mYXN0IDogICAgICAgNC41NzAgbWljcm9zL29wIDIxODgwNiBvcHMvc2Vj OyAoMTAwMDEwMCBvZg0KPiA+Pj4+IDEwMDAxMDAgZm91bmQsIGlzc3VlZCA0NjYzOSBub24tZXhp c3Qga2V5cykNCj4gPj4+Pg0KPiA+Pj4+ID09PT09PT09PT09PT09PT09PT09PT09PT09PQ0KPiA+ Pj4+IHRoZW4gSSBtb2RpZnkgcmVhZHJhbmRvbWZhc3QgdG8gdXNlIEl0ZXJhdG9yIEFQSVswXToN Cj4gPj4+Pg0KPiA+Pj4+IFtyb290QGh1bnRlci1ub2RlMiByb2Nrc2RiXSMgLi9kYl9iZW5jaCAt dXNlX2V4aXN0aW5nX2RiDQo+ID4+Pj4gLWJlbmNobWFya3MgcmVhZHJhbmRvbWZhc3QNCj4gPj4+ PiBMZXZlbERCOiAgICB2ZXJzaW9uIDQuMw0KPiA+Pj4+IERhdGU6ICAgICAgIFdlZCBKdW4gIDEg MDA6MzM6MDMgMjAxNg0KPiA+Pj4+IENQVTogICAgICAgIDMyICogSW50ZWwoUikgWGVvbihSKSBD UFUgRTUtMjY4MCAwIEAgMi43MEdIeg0KPiA+Pj4+IENQVUNhY2hlOiAgIDIwNDgwIEtCDQo+ID4+ Pj4gS2V5czogICAgICAgMTYgYnl0ZXMgZWFjaA0KPiA+Pj4+IFZhbHVlczogICAgIDEwMCBieXRl cyBlYWNoICg1MCBieXRlcyBhZnRlciBjb21wcmVzc2lvbikNCj4gPj4+PiBFbnRyaWVzOiAgICAx MDAwMDAwDQo+ID4+Pj4gUHJlZml4OiAgICAwIGJ5dGVzDQo+ID4+Pj4gS2V5cyBwZXIgcHJlZml4 OiAgICAwDQo+ID4+Pj4gUmF3U2l6ZTogICAgMTEwLjYgTUIgKGVzdGltYXRlZCkNCj4gPj4+PiBG aWxlU2l6ZTogICA2Mi45IE1CIChlc3RpbWF0ZWQpDQo+ID4+Pj4gV3JpdGVzIHBlciBzZWNvbmQ6 IDANCj4gPj4+PiBDb21wcmVzc2lvbjogU25hcHB5DQo+ID4+Pj4gTWVtdGFibGVyZXA6IHNraXBf bGlzdA0KPiA+Pj4+IFBlcmYgTGV2ZWw6IDANCj4gPj4+PiBXQVJOSU5HOiBBc3NlcnRpb25zIGFy ZSBlbmFibGVkOyBiZW5jaG1hcmtzIHVubmVjZXNzYXJpbHkgc2xvdw0KPiA+Pj4+IC0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPiA+Pj4+IERCIHBhdGg6 IFsvdG1wL3JvY2tzZGJ0ZXN0LTAvZGJiZW5jaF0NCj4gPj4+PiByZWFkcmFuZG9tZmFzdCA6ICAg ICAgNDUuMTg4IG1pY3Jvcy9vcCAyMjEyOSBvcHMvc2VjOyAoMTAwMDEwMCBvZg0KPiA+Pj4+IDEw MDAxMDAgZm91bmQsIGlzc3VlZCA0NjYzOSBub24tZXhpc3Qga2V5cykNCj4gPj4+Pg0KPiA+Pj4+ DQo+ID4+Pj4gNDUuMTggdXMvb3AgdnMgNC41N3VzL29wIQ0KPiA+Pj4+DQo+ID4+Pj4gVGhlIHRl c3QgY2FuIGJlIHJlcGVhdGVkIGFuZCBlYXN5IHRvIGRvISBQbHogY29ycmVjdCBpZiBJJ20gZG9p bmcNCj4gPj4+PiBmb29saXNoIHRoaW5nIEknbSBub3QgYXdhcmUuLg0KPiA+Pj4+DQo+ID4+Pj4g U28gSSBwcm9wb3NhbCB0aGlzIFBSOiBodHRwczovL2dpdGh1Yi5jb20vY2VwaC9jZXBoL3B1bGwv OTQxMQ0KPiA+Pj4+DQo+ID4+Pj4gV2Ugc3RpbGwgY2FuIG1ha2UgZnVydGhlciBpbXByb3ZlbWVu dHMgYnkgc2Nhbm5pbmcgYWxsIGl0ZXJhdGUNCj4gPj4+PiB1c2FnZSB0byBtYWtlIGl0IGJldHRl ciENCj4gPj4+Pg0KPiA+Pj4+IFswXToNCj4gPj4+PiAtLS0gYS9kYi9kYl9iZW5jaC5jYw0KPiA+ Pj4+ICsrKyBiL2RiL2RiX2JlbmNoLmNjDQo+ID4+Pj4gQEAgLTI5MjMsMTQgKzI5MjMsMTIgQEAg Y2xhc3MgQmVuY2htYXJrIHsNCj4gPj4+PiAgICAgICAgICBpbnQ2NF90IGtleV9yYW5kID0gdGhy ZWFkLT5yYW5kLk5leHQoKSAmIChwb3QgLSAxKTsNCj4gPj4+PiAgICAgICAgICBHZW5lcmF0ZUtl eUZyb21JbnQoa2V5X3JhbmQsIEZMQUdTX251bSwgJmtleSk7DQo+ID4+Pj4gICAgICAgICAgKyty ZWFkOw0KPiA+Pj4+IC0gICAgICAgIGF1dG8gc3RhdHVzID0gZGItPkdldChvcHRpb25zLCBrZXks ICZ2YWx1ZSk7DQo+ID4+Pj4gLSAgICAgICAgaWYgKHN0YXR1cy5vaygpKSB7DQo+ID4+Pj4gLSAg ICAgICAgICArK2ZvdW5kOw0KPiA+Pj4+IC0gICAgICAgIH0gZWxzZSBpZiAoIXN0YXR1cy5Jc05v dEZvdW5kKCkpIHsNCj4gPj4+PiAtICAgICAgICAgIGZwcmludGYoc3RkZXJyLCAiR2V0IHJldHVy bmVkIGFuIGVycm9yOiAlc1xuIiwNCj4gPj4+PiAtICAgICAgICAgICAgICAgICAgc3RhdHVzLlRv U3RyaW5nKCkuY19zdHIoKSk7DQo+ID4+Pj4gLSAgICAgICAgICBhYm9ydCgpOw0KPiA+Pj4+IC0g ICAgICAgIH0NCj4gPj4+PiArICAgICAgICBJdGVyYXRvciogaXRlciA9IGRiLT5OZXdJdGVyYXRv cihvcHRpb25zKTsNCj4gPj4+PiArICAgICAgaXRlci0+U2VlayhrZXkpOw0KPiA+Pj4+ICsgICAg ICBpZiAoaXRlci0+VmFsaWQoKSAmJiBpdGVyLT5rZXkoKS5jb21wYXJlKGtleSkgPT0gMCkgew0K PiA+Pj4+ICsgICAgICAgIGZvdW5kKys7DQo+ID4+Pj4gKyAgICAgIH0NCj4gPj4+PiArDQo+ID4+ Pj4gICAgICAgICAgaWYgKGtleV9yYW5kID49IEZMQUdTX251bSkgew0KPiA+Pj4+ICAgICAgICAg ICAgKytub25leGlzdDsNCj4gPj4+PiAgICAgICAgICB9DQo+ID4+Pj4gLS0NCj4gPj4+PiBUbyB1 bnN1YnNjcmliZSBmcm9tIHRoaXMgbGlzdDogc2VuZCB0aGUgbGluZSAidW5zdWJzY3JpYmUNCj4g Pj4+PiBjZXBoLWRldmVsIiBpbiB0aGUgYm9keSBvZiBhIG1lc3NhZ2UgdG8gbWFqb3Jkb21vQHZn ZXIua2VybmVsLm9yZw0KPiA+Pj4+IE1vcmUgbWFqb3Jkb21vIGluZm8gYXQgIGh0dHA6Ly92Z2Vy Lmtlcm5lbC5vcmcvbWFqb3Jkb21vLWluZm8uaHRtbA0KPiA+PiAtLQ0KPiA+PiBUbyB1bnN1YnNj cmliZSBmcm9tIHRoaXMgbGlzdDogc2VuZCB0aGUgbGluZSAidW5zdWJzY3JpYmUgY2VwaC1kZXZl bCINCj4gPj4gaW4gdGhlIGJvZHkgb2YgYSBtZXNzYWdlIHRvIG1ham9yZG9tb0B2Z2VyLmtlcm5l bC5vcmcgTW9yZQ0KPiBtYWpvcmRvbW8NCj4gPj4gaW5mbyBhdCAgaHR0cDovL3ZnZXIua2VybmVs Lm9yZy9tYWpvcmRvbW8taW5mby5odG1sDQo+IC0tDQo+IFRvIHVuc3Vic2NyaWJlIGZyb20gdGhp cyBsaXN0OiBzZW5kIHRoZSBsaW5lICJ1bnN1YnNjcmliZSBjZXBoLWRldmVsIiBpbiB0aGUNCj4g Ym9keSBvZiBhIG1lc3NhZ2UgdG8gbWFqb3Jkb21vQHZnZXIua2VybmVsLm9yZyBNb3JlIG1ham9y ZG9tbyBpbmZvIGF0DQo+IGh0dHA6Ly92Z2VyLmtlcm5lbC5vcmcvbWFqb3Jkb21vLWluZm8uaHRt bA0K -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
DQpPbiBUdWUsIE1heSAzMSwgMjAxNiBhdCA3OjM4IFBNLCBIYW9tYWkgV2FuZyA8aGFvbWFpQHhz a3kuY29tPiB3cm90ZToNCj4gT24gV2VkLCBKdW4gMSwgMjAxNiBhdCAzOjQwIEFNLCBKaWFuamlh biBIdW8gPGppYW5qaWFuLmh1b0BzYW1zdW5nLmNvbT4gd3JvdGU6DQo+Pg0KPj4gT24gVHVlLCBN YXkgMzEsIDIwMTYgYXQgMTE6MTIgQU0sIEhhb21haSBXYW5nIDxoYW9tYWlAeHNreS5jb20+IHdy b3RlOg0KPj4+IE9uIFdlZCwgSnVuIDEsIDIwMTYgYXQgMjowOCBBTSwgSmlhbmppYW4gSHVvIDxq aWFuamlhbi5odW9Ac2Ftc3VuZy5jb20+IHdyb3RlOg0KPj4+PiBIaSBIYW9tYWksDQo+Pj4+DQo+ Pj4+IEkgbm90aWNlZCB0aGlzIGFzIHdlbGwsIGFuZCBtYWRlIHNhbWUgY2hhbmdlcyB0byBSb2Nr c0RCU3RvcmUgaW4gdGhpcyBQUiBsYXN0IHdlZWs6DQo+Pj4+IGh0dHBzOi8vZ2l0aHViLmNvbS9j ZXBoL2NlcGgvcHVsbC85MjE1DQo+Pj4+DQo+Pj4+IE9uZSB0aGluZyB3aGljaCBpcyBldmVuIHdv cnNlLCAgc2VlayB3aWxsIGJ5cGFzcyByb3cgY2FjaGUsIHNvIGt2IHBhaXJzIHdvbid0IGJlIGFi bGUgdG8gYmUgY2FjaGVkIGluIHJvdyBjYWNoZS4NCj4+Pj4gSSBhbSB3b3JraW5nIHRvIGJlbmNo bWFyayB0aGUgcGVyZm9ybWFuY2UgaW1wYWN0LCB3aWxsIHB1Ymxpc2ggdGhlIHJlc3VsdHMgYWZ0 ZXIgSSBhbSBkb25lIHRoaXMgd2Vlay4NCj4+Pg0KPj4+IE9oLCBjb29sISBJIHRoaW5rIHlvdSBj YW4gY2hlcnJ5LXBpY2sgbXkgYW5vdGhlciBsZXZlbGRiIGZpeC4NCj4+DQo+PiBObyBwcm9ibGVt LCBJIHdpbGwgZG8gdGhhdC4NCj4+Pg0KPj4+IEJUVywgZG8geW91IHBheSBhbiBhdHRlbnRpb24g dG8gcHJlZml4IHNlZWsgYXBpPyBJIHRoaW5rIGl0IHdpbGwgYmUNCj4+PiBtb3JlIHN1aXRhYmxl IHRoYW4gY29sdW1uIGZhbWlseSBpbiBjZXBoIGNhc2UuIElmIHdlIGNhbiBoYXZlDQo+Pj4gd2Vs bC1kZWZpbmVkIHByZWZpeCBydWxlLCB3ZSBjYW4gbWFrZSBtb3N0IG9mIHJhbmdlIHF1ZXJ5IGNo ZWFwZXIhDQo+Pg0KPj4gV2hlbiBrZXlzIHdpdGggc2FtZSBwcmVmaXggYXJlIHN0b3JlZCBpbiB0 aGVpciBvd24gU1NUIGZpbGVzKENGIGNhc2UpLCBldmVuIHNlZWtpbmcgd2l0aG91dCBwcmVmaXgg d2lsbCBiZSBmYXN0ZXIgdGhhbiBzZWVraW5nIHdpdGggcHJlZml4IGJ1dCBtaXhlZCB3aXRoIGRp ZmZlcmVudCBwcmVmaXhlZCBrZXlzPw0KPg0KPiBGcm9tIG15IGN1cnJlbnQgdmlldywgcHJlZml4 IGNvdWxkIGJlIG1vcmUgZmxleGlibGUuIEZvciBleGFtcGxlLCBlYWNoDQo+IHJndyBidWNrZXQg aW5kZXggY291bGQgdXNlIG9uZSBwcmVmaXggdG8gbWFrZSBlYWNoIGJ1Y2tldCBpbmRleCBvYmpl Y3QNCj4gc2VlayBzZXBhcmF0ZWQuIEZvciBDRiwgaXQgd291bGQgYmUgdG9vIGhlYXZyeS4NCg0K U3VyZSwgbm8gZG91YnQgcHJlZml4IHNlZWsgd2lsbCBoZWxwIHNvbWUgY2FzZXMsIENGIHdpbGwg aGVscCBvdGhlcnMuIEFuZCBwcmVmaXggd2lsbCBtb3N0bHkgYmVuZWZpdCBzZWVrLCB3aGlsZSBD RiBoYXMgYnJvYWRlciBpbXBhY3QuDQo+DQo+PiBJIGFtIG5vdCBzdXJlIHdoYXQgb3B0aW1pemF0 aW9uIHByZWZpeCBzZWVrIHdpbGwgdXNlIGludGVybmFsbHkgZm9yIGJsb2NrIGJhc2VkIGZvcm1h dCwgYnV0IHRvIG1lLCBpdCdzIGhhcmQgdG8gYmVhdCB0aGUgY2FzZSB3aGVuIHlvdSBvbmx5IGhh dmUgb25lIHByZWZpeGVkIGtleXMgc3RvcmVkIHNlcGFyYXRlbHkuDQo+Pj4NCj4+Pj4NCj4+Pj4g SmlhbmppYW4NCj4+Pj4NCj4+Pj4gT24gVHVlLCBNYXkgMzEsIDIwMTYgYXQgOTo0OSBBTSwgSGFv bWFpIFdhbmcgPGhhb21haUB4c2t5LmNvbT4gd3JvdGU6DQo+Pj4+PiBIaSBTYWdlIGFuZCBNYXJr LA0KPj4+Pj4NCj4+Pj4+IEFzIG1lbnRpb25lZCBpbiBCbHVlU3RvcmUgc3RhbmR1cCwgSSBmb3Vu ZCByb2Nrc2RiIGl0ZXJhdG9yICpTZWVrKg0KPj4+Pj4gd29uJ3QgdXNlIGJsb29tIGZpbHRlciBs aWtlICpHZXQqLg0KPj4+Pj4NCj4+Pj4+ICpHZXQqIGltcGw6IGl0IHdpbGwgbG9vayBhdCBmaWx0 ZXIgZmlyc3RseQ0KPj4+Pj4gaHR0cHM6Ly9naXRodWIuY29tL2ZhY2Vib29rL3JvY2tzZGIvYmxv Yi9tYXN0ZXIvdGFibGUvYmxvY2tfYmFzZWRfdGFibGVfcmVhZGVyLmNjI0wxMzY5DQo+Pj4+Pg0K Pj4+Pj4gSXRlcmF0b3IgKlNlZWsqOiBpdCB3aWxsIGRvIGJpbmFyeSBzZWFyY2gsIGJ5IGRlZmF1 bHQgd2UgZG9uJ3Qgc3BlY2lmeQ0KPj4+Pj4gcHJlZml4IGZlYXR1cmUoaHR0cHM6Ly9naXRodWIu Y29tL2ZhY2Vib29rL3JvY2tzZGIvd2lraS9QcmVmaXgtU2Vlay1BUEktQ2hhbmdlcykuDQo+Pj4+ PiBodHRwczovL2dpdGh1Yi5jb20vZmFjZWJvb2svcm9ja3NkYi9ibG9iL21hc3Rlci90YWJsZS9i bG9jay5jYyNMOTQNCj4+Pj4+DQo+Pj4+PiBTbyBJIHVzZSBhIHNpbXBsZSB0ZXN0czoNCj4+Pj4+ DQo+Pj4+PiAuL2RiX2JlbmNoIC1udW0gMTAwMDAwMDAgIC1iZW5jaG1hcmtzIGZpbGxiYXRjaA0K Pj4+Pj4gZmlsbCB0aGUgZGIgZmlyc3RseSB3aXRoIDEwMDB3IHJlY29yZHMuDQo+Pj4+Pg0KPj4+ Pj4gLi9kYl9iZW5jaCAtdXNlX2V4aXN0aW5nX2RiICAtYmVuY2htYXJrcyByZWFkcmFuZG9tZmFz dA0KPj4+Pj4gcmVhZHJhbmRvbWZhc3QgY2FzZSB3aWxsIHVzZSAqR2V0KiBBUEkgdG8gcmV0cml2 ZSBkYXRhDQo+Pj4+Pg0KPj4+Pj4gW3Jvb3RAaHVudGVyLW5vZGUyIHJvY2tzZGJdIyAuL2RiX2Jl bmNoIC11c2VfZXhpc3RpbmdfZGIgIC1iZW5jaG1hcmtzDQo+Pj4+PiByZWFkcmFuZG9tZmFzdA0K Pj4+Pj4NCj4+Pj4+IExldmVsREI6ICAgIHZlcnNpb24gNC4zDQo+Pj4+PiBEYXRlOiAgICAgICBX ZWQgSnVuICAxIDAwOjI5OjE2IDIwMTYNCj4+Pj4+IENQVTogICAgICAgIDMyICogSW50ZWwoUikg WGVvbihSKSBDUFUgRTUtMjY4MCAwIEAgMi43MEdIeg0KPj4+Pj4gQ1BVQ2FjaGU6ICAgMjA0ODAg S0INCj4+Pj4+IEtleXM6ICAgICAgIDE2IGJ5dGVzIGVhY2gNCj4+Pj4+IFZhbHVlczogICAgIDEw MCBieXRlcyBlYWNoICg1MCBieXRlcyBhZnRlciBjb21wcmVzc2lvbikNCj4+Pj4+IEVudHJpZXM6 ICAgIDEwMDAwMDANCj4+Pj4+IFByZWZpeDogICAgMCBieXRlcw0KPj4+Pj4gS2V5cyBwZXIgcHJl Zml4OiAgICAwDQo+Pj4+PiBSYXdTaXplOiAgICAxMTAuNiBNQiAoZXN0aW1hdGVkKQ0KPj4+Pj4g RmlsZVNpemU6ICAgNjIuOSBNQiAoZXN0aW1hdGVkKQ0KPj4+Pj4gV3JpdGVzIHBlciBzZWNvbmQ6 IDANCj4+Pj4+IENvbXByZXNzaW9uOiBTbmFwcHkNCj4+Pj4+IE1lbXRhYmxlcmVwOiBza2lwX2xp c3QNCj4+Pj4+IFBlcmYgTGV2ZWw6IDANCj4+Pj4+IFdBUk5JTkc6IEFzc2VydGlvbnMgYXJlIGVu YWJsZWQ7IGJlbmNobWFya3MgdW5uZWNlc3NhcmlseSBzbG93DQo+Pj4+PiAtLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCj4+Pj4+IERCIHBhdGg6IFsvdG1w L3JvY2tzZGJ0ZXN0LTAvZGJiZW5jaF0NCj4+Pj4+IHJlYWRyYW5kb21mYXN0IDogICAgICAgNC41 NzAgbWljcm9zL29wIDIxODgwNiBvcHMvc2VjOyAoMTAwMDEwMCBvZg0KPj4+Pj4gMTAwMDEwMCBm b3VuZCwgaXNzdWVkIDQ2NjM5IG5vbi1leGlzdCBrZXlzKQ0KPj4+Pj4NCj4+Pj4+ID09PT09PT09 PT09PT09PT09PT09PT09PT09PQ0KPj4+Pj4gdGhlbiBJIG1vZGlmeSByZWFkcmFuZG9tZmFzdCB0 byB1c2UgSXRlcmF0b3IgQVBJWzBdOg0KPj4+Pj4NCj4+Pj4+IFtyb290QGh1bnRlci1ub2RlMiBy b2Nrc2RiXSMgLi9kYl9iZW5jaCAtdXNlX2V4aXN0aW5nX2RiICAtYmVuY2htYXJrcw0KPj4+Pj4g cmVhZHJhbmRvbWZhc3QNCj4+Pj4+IExldmVsREI6ICAgIHZlcnNpb24gNC4zDQo+Pj4+PiBEYXRl OiAgICAgICBXZWQgSnVuICAxIDAwOjMzOjAzIDIwMTYNCj4+Pj4+IENQVTogICAgICAgIDMyICog SW50ZWwoUikgWGVvbihSKSBDUFUgRTUtMjY4MCAwIEAgMi43MEdIeg0KPj4+Pj4gQ1BVQ2FjaGU6 ICAgMjA0ODAgS0INCj4+Pj4+IEtleXM6ICAgICAgIDE2IGJ5dGVzIGVhY2gNCj4+Pj4+IFZhbHVl czogICAgIDEwMCBieXRlcyBlYWNoICg1MCBieXRlcyBhZnRlciBjb21wcmVzc2lvbikNCj4+Pj4+ IEVudHJpZXM6ICAgIDEwMDAwMDANCj4+Pj4+IFByZWZpeDogICAgMCBieXRlcw0KPj4+Pj4gS2V5 cyBwZXIgcHJlZml4OiAgICAwDQo+Pj4+PiBSYXdTaXplOiAgICAxMTAuNiBNQiAoZXN0aW1hdGVk KQ0KPj4+Pj4gRmlsZVNpemU6ICAgNjIuOSBNQiAoZXN0aW1hdGVkKQ0KPj4+Pj4gV3JpdGVzIHBl ciBzZWNvbmQ6IDANCj4+Pj4+IENvbXByZXNzaW9uOiBTbmFwcHkNCj4+Pj4+IE1lbXRhYmxlcmVw OiBza2lwX2xpc3QNCj4+Pj4+IFBlcmYgTGV2ZWw6IDANCj4+Pj4+IFdBUk5JTkc6IEFzc2VydGlv bnMgYXJlIGVuYWJsZWQ7IGJlbmNobWFya3MgdW5uZWNlc3NhcmlseSBzbG93DQo+Pj4+PiAtLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCj4+Pj4+IERCIHBh dGg6IFsvdG1wL3JvY2tzZGJ0ZXN0LTAvZGJiZW5jaF0NCj4+Pj4+IHJlYWRyYW5kb21mYXN0IDog ICAgICA0NS4xODggbWljcm9zL29wIDIyMTI5IG9wcy9zZWM7ICgxMDAwMTAwIG9mDQo+Pj4+PiAx MDAwMTAwIGZvdW5kLCBpc3N1ZWQgNDY2Mzkgbm9uLWV4aXN0IGtleXMpDQo+Pj4+Pg0KPj4+Pj4N Cj4+Pj4+IDQ1LjE4IHVzL29wIHZzIDQuNTd1cy9vcCENCj4+Pj4+DQo+Pj4+PiBUaGUgdGVzdCBj YW4gYmUgcmVwZWF0ZWQgYW5kIGVhc3kgdG8gZG8hIFBseiBjb3JyZWN0IGlmIEknbSBkb2luZw0K Pj4+Pj4gZm9vbGlzaCB0aGluZyBJJ20gbm90IGF3YXJlLi4NCj4+Pj4+DQo+Pj4+PiBTbyBJIHBy b3Bvc2FsIHRoaXMgUFI6IGh0dHBzOi8vZ2l0aHViLmNvbS9jZXBoL2NlcGgvcHVsbC85NDExDQo+ Pj4+Pg0KPj4+Pj4gV2Ugc3RpbGwgY2FuIG1ha2UgZnVydGhlciBpbXByb3ZlbWVudHMgYnkgc2Nh bm5pbmcgYWxsIGl0ZXJhdGUgdXNhZ2UNCj4+Pj4+IHRvIG1ha2UgaXQgYmV0dGVyIQ0KPj4+Pj4N Cj4+Pj4+IFswXToNCj4+Pj4+IC0tLSBhL2RiL2RiX2JlbmNoLmNjDQo+Pj4+PiArKysgYi9kYi9k Yl9iZW5jaC5jYw0KPj4+Pj4gQEAgLTI5MjMsMTQgKzI5MjMsMTIgQEAgY2xhc3MgQmVuY2htYXJr IHsNCj4+Pj4+ICAgICAgICAgIGludDY0X3Qga2V5X3JhbmQgPSB0aHJlYWQtPnJhbmQuTmV4dCgp ICYgKHBvdCAtIDEpOw0KPj4+Pj4gICAgICAgICAgR2VuZXJhdGVLZXlGcm9tSW50KGtleV9yYW5k LCBGTEFHU19udW0sICZrZXkpOw0KPj4+Pj4gICAgICAgICAgKytyZWFkOw0KPj4+Pj4gLSAgICAg ICAgYXV0byBzdGF0dXMgPSBkYi0+R2V0KG9wdGlvbnMsIGtleSwgJnZhbHVlKTsNCj4+Pj4+IC0g ICAgICAgIGlmIChzdGF0dXMub2soKSkgew0KPj4+Pj4gLSAgICAgICAgICArK2ZvdW5kOw0KPj4+ Pj4gLSAgICAgICAgfSBlbHNlIGlmICghc3RhdHVzLklzTm90Rm91bmQoKSkgew0KPj4+Pj4gLSAg ICAgICAgICBmcHJpbnRmKHN0ZGVyciwgIkdldCByZXR1cm5lZCBhbiBlcnJvcjogJXNcbiIsDQo+ Pj4+PiAtICAgICAgICAgICAgICAgICAgc3RhdHVzLlRvU3RyaW5nKCkuY19zdHIoKSk7DQo+Pj4+ PiAtICAgICAgICAgIGFib3J0KCk7DQo+Pj4+PiAtICAgICAgICB9DQo+Pj4+PiArICAgICAgICBJ dGVyYXRvciogaXRlciA9IGRiLT5OZXdJdGVyYXRvcihvcHRpb25zKTsNCj4+Pj4+ICsgICAgICBp dGVyLT5TZWVrKGtleSk7DQo+Pj4+PiArICAgICAgaWYgKGl0ZXItPlZhbGlkKCkgJiYgaXRlci0+ a2V5KCkuY29tcGFyZShrZXkpID09IDApIHsNCj4+Pj4+ICsgICAgICAgIGZvdW5kKys7DQo+Pj4+ PiArICAgICAgfQ0KPj4+Pj4gKw0KPj4+Pj4gICAgICAgICAgaWYgKGtleV9yYW5kID49IEZMQUdT X251bSkgew0KPj4+Pj4gICAgICAgICAgICArK25vbmV4aXN0Ow0KPj4+Pj4gICAgICAgICAgfQ0K Pj4+Pj4gLS0NCj4+Pj4+IFRvIHVuc3Vic2NyaWJlIGZyb20gdGhpcyBsaXN0OiBzZW5kIHRoZSBs aW5lICJ1bnN1YnNjcmliZSBjZXBoLWRldmVsIiBpbg0KPj4+Pj4gdGhlIGJvZHkgb2YgYSBtZXNz YWdlIHRvIG1ham9yZG9tb0B2Z2VyLmtlcm5lbC5vcmcNCj4+Pj4+IE1vcmUgbWFqb3Jkb21vIGlu Zm8gYXQgIGh0dHA6Ly92Z2VyLmtlcm5lbC5vcmcvbWFqb3Jkb21vLWluZm8uaHRtbA0KPj4+IC0t DQo+Pj4gVG8gdW5zdWJzY3JpYmUgZnJvbSB0aGlzIGxpc3Q6IHNlbmQgdGhlIGxpbmUgInVuc3Vi c2NyaWJlIGNlcGgtZGV2ZWwiIGluDQo+Pj4gdGhlIGJvZHkgb2YgYSBtZXNzYWdlIHRvIG1ham9y ZG9tb0B2Z2VyLmtlcm5lbC5vcmcNCj4+PiBNb3JlIG1ham9yZG9tbyBpbmZvIGF0ICBodHRwOi8v dmdlci5rZXJuZWwub3JnL21ham9yZG9tby1pbmZvLmh0bWwNCj4gLS0NCj4gVG8gdW5zdWJzY3Jp YmUgZnJvbSB0aGlzIGxpc3Q6IHNlbmQgdGhlIGxpbmUgInVuc3Vic2NyaWJlIGNlcGgtZGV2ZWwi IGluDQo+IHRoZSBib2R5IG9mIGEgbWVzc2FnZSB0byBtYWpvcmRvbW9Admdlci5rZXJuZWwub3Jn DQo+IE1vcmUgbWFqb3Jkb21vIGluZm8gYXQgIGh0dHA6Ly92Z2VyLmtlcm5lbC5vcmcvbWFqb3Jk b21vLWluZm8uaHRtbA0K -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
SGkgQWxsZW4sDQoNCk9uIFR1ZSwgTWF5IDMxLCAyMDE2IGF0IDc6NTIgUE0sIEFsbGVuIFNhbXVl bHMgPEFsbGVuLlNhbXVlbHNAc2FuZGlzay5jb20+IHdyb3RlOg0KPj4gLS0tLS1PcmlnaW5hbCBN ZXNzYWdlLS0tLS0NCj4+IEZyb206IGNlcGgtZGV2ZWwtb3duZXJAdmdlci5rZXJuZWwub3JnIFtt YWlsdG86Y2VwaC1kZXZlbC0NCj4+IG93bmVyQHZnZXIua2VybmVsLm9yZ10gT24gQmVoYWxmIE9m IEhhb21haSBXYW5nDQo+PiBTZW50OiBUdWVzZGF5LCBNYXkgMzEsIDIwMTYgNzozOCBQTQ0KPj4g VG86IEppYW5qaWFuIEh1byA8amlhbmppYW4uaHVvQHNhbXN1bmcuY29tPg0KPj4gQ2M6IFNhZ2Ug V2VpbCA8c3dlaWxAcmVkaGF0LmNvbT47IE1hcmsgTmVsc29uIDxtbmVsc29uQHJlZGhhdC5jb20+ Ow0KPj4gY2VwaC1kZXZlbEB2Z2VyLmtlcm5lbC5vcmcNCj4+IFN1YmplY3Q6IFJlOiBSb2Nrc0RC IEluY29ycmVjdCBBUEkgVXNhZ2UNCj4+DQo+PiBPbiBXZWQsIEp1biAxLCAyMDE2IGF0IDM6NDAg QU0sIEppYW5qaWFuIEh1byA8amlhbmppYW4uaHVvQHNhbXN1bmcuY29tPg0KPj4gd3JvdGU6DQo+ PiA+DQo+PiA+IE9uIFR1ZSwgTWF5IDMxLCAyMDE2IGF0IDExOjEyIEFNLCBIYW9tYWkgV2FuZyA8 aGFvbWFpQHhza3kuY29tPg0KPj4gd3JvdGU6DQo+PiA+PiBPbiBXZWQsIEp1biAxLCAyMDE2IGF0 IDI6MDggQU0sIEppYW5qaWFuIEh1bw0KPj4gPGppYW5qaWFuLmh1b0BzYW1zdW5nLmNvbT4gd3Jv dGU6DQo+PiA+Pj4gSGkgSGFvbWFpLA0KPj4gPj4+DQo+PiA+Pj4gSSBub3RpY2VkIHRoaXMgYXMg d2VsbCwgYW5kIG1hZGUgc2FtZSBjaGFuZ2VzIHRvIFJvY2tzREJTdG9yZSBpbiB0aGlzIFBSDQo+ PiBsYXN0IHdlZWs6DQo+PiA+Pj4gaHR0cHM6Ly9naXRodWIuY29tL2NlcGgvY2VwaC9wdWxsLzky MTUNCj4+ID4+Pg0KPj4gPj4+IE9uZSB0aGluZyB3aGljaCBpcyBldmVuIHdvcnNlLCAgc2VlayB3 aWxsIGJ5cGFzcyByb3cgY2FjaGUsIHNvIGt2IHBhaXJzDQo+PiB3b24ndCBiZSBhYmxlIHRvIGJl IGNhY2hlZCBpbiByb3cgY2FjaGUuDQo+PiA+Pj4gSSBhbSB3b3JraW5nIHRvIGJlbmNobWFyayB0 aGUgcGVyZm9ybWFuY2UgaW1wYWN0LCB3aWxsIHB1Ymxpc2ggdGhlDQo+PiByZXN1bHRzIGFmdGVy IEkgYW0gZG9uZSB0aGlzIHdlZWsuDQo+PiA+Pg0KPj4gPj4gT2gsIGNvb2whIEkgdGhpbmsgeW91 IGNhbiBjaGVycnktcGljayBteSBhbm90aGVyIGxldmVsZGIgZml4Lg0KPj4gPg0KPj4gPiBObyBw cm9ibGVtLCBJIHdpbGwgZG8gdGhhdC4NCj4+ID4+DQo+PiA+PiBCVFcsIGRvIHlvdSBwYXkgYW4g YXR0ZW50aW9uIHRvIHByZWZpeCBzZWVrIGFwaT8gSSB0aGluayBpdCB3aWxsIGJlDQo+PiA+PiBt b3JlIHN1aXRhYmxlIHRoYW4gY29sdW1uIGZhbWlseSBpbiBjZXBoIGNhc2UuIElmIHdlIGNhbiBo YXZlDQo+PiA+PiB3ZWxsLWRlZmluZWQgcHJlZml4IHJ1bGUsIHdlIGNhbiBtYWtlIG1vc3Qgb2Yg cmFuZ2UgcXVlcnkgY2hlYXBlciENCj4+ID4NCj4+ID4gV2hlbiBrZXlzIHdpdGggc2FtZSBwcmVm aXggYXJlIHN0b3JlZCBpbiB0aGVpciBvd24gU1NUIGZpbGVzKENGIGNhc2UpLA0KPj4gZXZlbiBz ZWVraW5nIHdpdGhvdXQgcHJlZml4IHdpbGwgYmUgZmFzdGVyIHRoYW4gc2Vla2luZyB3aXRoIHBy ZWZpeCBidXQgbWl4ZWQNCj4+IHdpdGggZGlmZmVyZW50IHByZWZpeGVkIGtleXM/DQo+Pg0KPj4g RnJvbSBteSBjdXJyZW50IHZpZXcsIHByZWZpeCBjb3VsZCBiZSBtb3JlIGZsZXhpYmxlLiBGb3Ig ZXhhbXBsZSwgZWFjaCByZ3cNCj4+IGJ1Y2tldCBpbmRleCBjb3VsZCB1c2Ugb25lIHByZWZpeCB0 byBtYWtlIGVhY2ggYnVja2V0IGluZGV4IG9iamVjdCBzZWVrDQo+PiBzZXBhcmF0ZWQuIEZvciBD RiwgaXQgd291bGQgYmUgdG9vIGhlYXZyeS4NCj4NCj4gVGhpcyBtaWdodCBjYXVzZSBvdGhlciBw cm9ibGVtcy4gV291bGQgaXQgZHJhbWF0aWNhbGx5IGluY3JlYXNlIHRoZSBudW1iZXIgb2YgZmls ZXMgdGhhdCBCbHVlRlMgbmVlZHMgdG8gbWFuYWdlPyBJZiBzbywgdGhhdCBtaWdodCBlZmZlY3Rp dmVseSBicmVhayB0aGF0IGNvZGUgdG9vIChvZiBjb3Vyc2UgaXQncyBmaXhhYmxlIGFsc28gOikp DQoNClRoYW5rcyBmb3IgYnJpbmdpbmcgdXAgdGhpcyBpc3N1ZS4gV2hhdCdzIGN1cnJlbnQgbGlt aXQgb2YgbnVtYmVyIG9mIGZpbGVzIGZvciBCbHVlRlMsIGZyb20geW91ciBleHBlcmllbmNlPyAN CkppYW5qaWFuDQo+Pg0KPj4gPiBJIGFtIG5vdCBzdXJlIHdoYXQgb3B0aW1pemF0aW9uIHByZWZp eCBzZWVrIHdpbGwgdXNlIGludGVybmFsbHkgZm9yIGJsb2NrDQo+PiBiYXNlZCBmb3JtYXQsIGJ1 dCB0byBtZSwgaXQncyBoYXJkIHRvIGJlYXQgdGhlIGNhc2Ugd2hlbiB5b3Ugb25seSBoYXZlIG9u ZQ0KPj4gcHJlZml4ZWQga2V5cyBzdG9yZWQgc2VwYXJhdGVseS4NCj4+ID4+DQo+PiA+Pj4NCj4+ ID4+PiBKaWFuamlhbg0KPj4gPj4+DQo+PiA+Pj4gT24gVHVlLCBNYXkgMzEsIDIwMTYgYXQgOTo0 OSBBTSwgSGFvbWFpIFdhbmcgPGhhb21haUB4c2t5LmNvbT4NCj4+IHdyb3RlOg0KPj4gPj4+PiBI aSBTYWdlIGFuZCBNYXJrLA0KPj4gPj4+Pg0KPj4gPj4+PiBBcyBtZW50aW9uZWQgaW4gQmx1ZVN0 b3JlIHN0YW5kdXAsIEkgZm91bmQgcm9ja3NkYiBpdGVyYXRvciAqU2VlayoNCj4+ID4+Pj4gd29u J3QgdXNlIGJsb29tIGZpbHRlciBsaWtlICpHZXQqLg0KPj4gPj4+Pg0KPj4gPj4+PiAqR2V0KiBp bXBsOiBpdCB3aWxsIGxvb2sgYXQgZmlsdGVyIGZpcnN0bHkNCj4+ID4+Pj4NCj4+IGh0dHBzOi8v Z2l0aHViLmNvbS9mYWNlYm9vay9yb2Nrc2RiL2Jsb2IvbWFzdGVyL3RhYmxlL2Jsb2NrX2Jhc2Vk X3QNCj4+ID4+Pj4gYWJsZV9yZWFkZXIuY2MjTDEzNjkNCj4+ID4+Pj4NCj4+ID4+Pj4gSXRlcmF0 b3IgKlNlZWsqOiBpdCB3aWxsIGRvIGJpbmFyeSBzZWFyY2gsIGJ5IGRlZmF1bHQgd2UgZG9uJ3QN Cj4+ID4+Pj4gc3BlY2lmeSBwcmVmaXgNCj4+IGZlYXR1cmUoaHR0cHM6Ly9naXRodWIuY29tL2Zh Y2Vib29rL3JvY2tzZGIvd2lraS9QcmVmaXgtU2Vlay1BUEktDQo+PiBDaGFuZ2VzKS4NCj4+ID4+ Pj4gaHR0cHM6Ly9naXRodWIuY29tL2ZhY2Vib29rL3JvY2tzZGIvYmxvYi9tYXN0ZXIvdGFibGUv YmxvY2suY2MjTDk0DQo+PiA+Pj4+DQo+PiA+Pj4+IFNvIEkgdXNlIGEgc2ltcGxlIHRlc3RzOg0K Pj4gPj4+Pg0KPj4gPj4+PiAuL2RiX2JlbmNoIC1udW0gMTAwMDAwMDAgIC1iZW5jaG1hcmtzIGZp bGxiYXRjaCBmaWxsIHRoZSBkYiBmaXJzdGx5DQo+PiA+Pj4+IHdpdGggMTAwMHcgcmVjb3Jkcy4N Cj4+ID4+Pj4NCj4+ID4+Pj4gLi9kYl9iZW5jaCAtdXNlX2V4aXN0aW5nX2RiICAtYmVuY2htYXJr cyByZWFkcmFuZG9tZmFzdA0KPj4gPj4+PiByZWFkcmFuZG9tZmFzdCBjYXNlIHdpbGwgdXNlICpH ZXQqIEFQSSB0byByZXRyaXZlIGRhdGENCj4+ID4+Pj4NCj4+ID4+Pj4gW3Jvb3RAaHVudGVyLW5v ZGUyIHJvY2tzZGJdIyAuL2RiX2JlbmNoIC11c2VfZXhpc3RpbmdfZGINCj4+ID4+Pj4gLWJlbmNo bWFya3MgcmVhZHJhbmRvbWZhc3QNCj4+ID4+Pj4NCj4+ID4+Pj4gTGV2ZWxEQjogICAgdmVyc2lv biA0LjMNCj4+ID4+Pj4gRGF0ZTogICAgICAgV2VkIEp1biAgMSAwMDoyOToxNiAyMDE2DQo+PiA+ Pj4+IENQVTogICAgICAgIDMyICogSW50ZWwoUikgWGVvbihSKSBDUFUgRTUtMjY4MCAwIEAgMi43 MEdIeg0KPj4gPj4+PiBDUFVDYWNoZTogICAyMDQ4MCBLQg0KPj4gPj4+PiBLZXlzOiAgICAgICAx NiBieXRlcyBlYWNoDQo+PiA+Pj4+IFZhbHVlczogICAgIDEwMCBieXRlcyBlYWNoICg1MCBieXRl cyBhZnRlciBjb21wcmVzc2lvbikNCj4+ID4+Pj4gRW50cmllczogICAgMTAwMDAwMA0KPj4gPj4+ PiBQcmVmaXg6ICAgIDAgYnl0ZXMNCj4+ID4+Pj4gS2V5cyBwZXIgcHJlZml4OiAgICAwDQo+PiA+ Pj4+IFJhd1NpemU6ICAgIDExMC42IE1CIChlc3RpbWF0ZWQpDQo+PiA+Pj4+IEZpbGVTaXplOiAg IDYyLjkgTUIgKGVzdGltYXRlZCkNCj4+ID4+Pj4gV3JpdGVzIHBlciBzZWNvbmQ6IDANCj4+ID4+ Pj4gQ29tcHJlc3Npb246IFNuYXBweQ0KPj4gPj4+PiBNZW10YWJsZXJlcDogc2tpcF9saXN0DQo+ PiA+Pj4+IFBlcmYgTGV2ZWw6IDANCj4+ID4+Pj4gV0FSTklORzogQXNzZXJ0aW9ucyBhcmUgZW5h YmxlZDsgYmVuY2htYXJrcyB1bm5lY2Vzc2FyaWx5IHNsb3cNCj4+ID4+Pj4gLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQo+PiA+Pj4+IERCIHBhdGg6IFsv dG1wL3JvY2tzZGJ0ZXN0LTAvZGJiZW5jaF0NCj4+ID4+Pj4gcmVhZHJhbmRvbWZhc3QgOiAgICAg ICA0LjU3MCBtaWNyb3Mvb3AgMjE4ODA2IG9wcy9zZWM7ICgxMDAwMTAwIG9mDQo+PiA+Pj4+IDEw MDAxMDAgZm91bmQsIGlzc3VlZCA0NjYzOSBub24tZXhpc3Qga2V5cykNCj4+ID4+Pj4NCj4+ID4+ Pj4gPT09PT09PT09PT09PT09PT09PT09PT09PT09DQo+PiA+Pj4+IHRoZW4gSSBtb2RpZnkgcmVh ZHJhbmRvbWZhc3QgdG8gdXNlIEl0ZXJhdG9yIEFQSVswXToNCj4+ID4+Pj4NCj4+ID4+Pj4gW3Jv b3RAaHVudGVyLW5vZGUyIHJvY2tzZGJdIyAuL2RiX2JlbmNoIC11c2VfZXhpc3RpbmdfZGINCj4+ ID4+Pj4gLWJlbmNobWFya3MgcmVhZHJhbmRvbWZhc3QNCj4+ID4+Pj4gTGV2ZWxEQjogICAgdmVy c2lvbiA0LjMNCj4+ID4+Pj4gRGF0ZTogICAgICAgV2VkIEp1biAgMSAwMDozMzowMyAyMDE2DQo+ PiA+Pj4+IENQVTogICAgICAgIDMyICogSW50ZWwoUikgWGVvbihSKSBDUFUgRTUtMjY4MCAwIEAg Mi43MEdIeg0KPj4gPj4+PiBDUFVDYWNoZTogICAyMDQ4MCBLQg0KPj4gPj4+PiBLZXlzOiAgICAg ICAxNiBieXRlcyBlYWNoDQo+PiA+Pj4+IFZhbHVlczogICAgIDEwMCBieXRlcyBlYWNoICg1MCBi eXRlcyBhZnRlciBjb21wcmVzc2lvbikNCj4+ID4+Pj4gRW50cmllczogICAgMTAwMDAwMA0KPj4g Pj4+PiBQcmVmaXg6ICAgIDAgYnl0ZXMNCj4+ID4+Pj4gS2V5cyBwZXIgcHJlZml4OiAgICAwDQo+ PiA+Pj4+IFJhd1NpemU6ICAgIDExMC42IE1CIChlc3RpbWF0ZWQpDQo+PiA+Pj4+IEZpbGVTaXpl OiAgIDYyLjkgTUIgKGVzdGltYXRlZCkNCj4+ID4+Pj4gV3JpdGVzIHBlciBzZWNvbmQ6IDANCj4+ ID4+Pj4gQ29tcHJlc3Npb246IFNuYXBweQ0KPj4gPj4+PiBNZW10YWJsZXJlcDogc2tpcF9saXN0 DQo+PiA+Pj4+IFBlcmYgTGV2ZWw6IDANCj4+ID4+Pj4gV0FSTklORzogQXNzZXJ0aW9ucyBhcmUg ZW5hYmxlZDsgYmVuY2htYXJrcyB1bm5lY2Vzc2FyaWx5IHNsb3cNCj4+ID4+Pj4gLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQo+PiA+Pj4+IERCIHBhdGg6 IFsvdG1wL3JvY2tzZGJ0ZXN0LTAvZGJiZW5jaF0NCj4+ID4+Pj4gcmVhZHJhbmRvbWZhc3QgOiAg ICAgIDQ1LjE4OCBtaWNyb3Mvb3AgMjIxMjkgb3BzL3NlYzsgKDEwMDAxMDAgb2YNCj4+ID4+Pj4g MTAwMDEwMCBmb3VuZCwgaXNzdWVkIDQ2NjM5IG5vbi1leGlzdCBrZXlzKQ0KPj4gPj4+Pg0KPj4g Pj4+Pg0KPj4gPj4+PiA0NS4xOCB1cy9vcCB2cyA0LjU3dXMvb3AhDQo+PiA+Pj4+DQo+PiA+Pj4+ IFRoZSB0ZXN0IGNhbiBiZSByZXBlYXRlZCBhbmQgZWFzeSB0byBkbyEgUGx6IGNvcnJlY3QgaWYg SSdtIGRvaW5nDQo+PiA+Pj4+IGZvb2xpc2ggdGhpbmcgSSdtIG5vdCBhd2FyZS4uDQo+PiA+Pj4+ DQo+PiA+Pj4+IFNvIEkgcHJvcG9zYWwgdGhpcyBQUjogaHR0cHM6Ly9naXRodWIuY29tL2NlcGgv Y2VwaC9wdWxsLzk0MTENCj4+ID4+Pj4NCj4+ID4+Pj4gV2Ugc3RpbGwgY2FuIG1ha2UgZnVydGhl ciBpbXByb3ZlbWVudHMgYnkgc2Nhbm5pbmcgYWxsIGl0ZXJhdGUNCj4+ID4+Pj4gdXNhZ2UgdG8g bWFrZSBpdCBiZXR0ZXIhDQo+PiA+Pj4+DQo+PiA+Pj4+IFswXToNCj4+ID4+Pj4gLS0tIGEvZGIv ZGJfYmVuY2guY2MNCj4+ID4+Pj4gKysrIGIvZGIvZGJfYmVuY2guY2MNCj4+ID4+Pj4gQEAgLTI5 MjMsMTQgKzI5MjMsMTIgQEAgY2xhc3MgQmVuY2htYXJrIHsNCj4+ID4+Pj4gICAgICAgICAgaW50 NjRfdCBrZXlfcmFuZCA9IHRocmVhZC0+cmFuZC5OZXh0KCkgJiAocG90IC0gMSk7DQo+PiA+Pj4+ ICAgICAgICAgIEdlbmVyYXRlS2V5RnJvbUludChrZXlfcmFuZCwgRkxBR1NfbnVtLCAma2V5KTsN Cj4+ID4+Pj4gICAgICAgICAgKytyZWFkOw0KPj4gPj4+PiAtICAgICAgICBhdXRvIHN0YXR1cyA9 IGRiLT5HZXQob3B0aW9ucywga2V5LCAmdmFsdWUpOw0KPj4gPj4+PiAtICAgICAgICBpZiAoc3Rh dHVzLm9rKCkpIHsNCj4+ID4+Pj4gLSAgICAgICAgICArK2ZvdW5kOw0KPj4gPj4+PiAtICAgICAg ICB9IGVsc2UgaWYgKCFzdGF0dXMuSXNOb3RGb3VuZCgpKSB7DQo+PiA+Pj4+IC0gICAgICAgICAg ZnByaW50ZihzdGRlcnIsICJHZXQgcmV0dXJuZWQgYW4gZXJyb3I6ICVzXG4iLA0KPj4gPj4+PiAt ICAgICAgICAgICAgICAgICAgc3RhdHVzLlRvU3RyaW5nKCkuY19zdHIoKSk7DQo+PiA+Pj4+IC0g ICAgICAgICAgYWJvcnQoKTsNCj4+ID4+Pj4gLSAgICAgICAgfQ0KPj4gPj4+PiArICAgICAgICBJ dGVyYXRvciogaXRlciA9IGRiLT5OZXdJdGVyYXRvcihvcHRpb25zKTsNCj4+ID4+Pj4gKyAgICAg IGl0ZXItPlNlZWsoa2V5KTsNCj4+ID4+Pj4gKyAgICAgIGlmIChpdGVyLT5WYWxpZCgpICYmIGl0 ZXItPmtleSgpLmNvbXBhcmUoa2V5KSA9PSAwKSB7DQo+PiA+Pj4+ICsgICAgICAgIGZvdW5kKys7 DQo+PiA+Pj4+ICsgICAgICB9DQo+PiA+Pj4+ICsNCj4+ID4+Pj4gICAgICAgICAgaWYgKGtleV9y YW5kID49IEZMQUdTX251bSkgew0KPj4gPj4+PiAgICAgICAgICAgICsrbm9uZXhpc3Q7DQo+PiA+ Pj4+ICAgICAgICAgIH0NCj4+ID4+Pj4gLS0NCj4+ID4+Pj4gVG8gdW5zdWJzY3JpYmUgZnJvbSB0 aGlzIGxpc3Q6IHNlbmQgdGhlIGxpbmUgInVuc3Vic2NyaWJlDQo+PiA+Pj4+IGNlcGgtZGV2ZWwi IGluIHRoZSBib2R5IG9mIGEgbWVzc2FnZSB0byBtYWpvcmRvbW9Admdlci5rZXJuZWwub3JnDQo+ PiA+Pj4+IE1vcmUgbWFqb3Jkb21vIGluZm8gYXQgIGh0dHA6Ly92Z2VyLmtlcm5lbC5vcmcvbWFq b3Jkb21vLWluZm8uaHRtbA0KPj4gPj4gLS0NCj4+ID4+IFRvIHVuc3Vic2NyaWJlIGZyb20gdGhp cyBsaXN0OiBzZW5kIHRoZSBsaW5lICJ1bnN1YnNjcmliZSBjZXBoLWRldmVsIg0KPj4gPj4gaW4g dGhlIGJvZHkgb2YgYSBtZXNzYWdlIHRvIG1ham9yZG9tb0B2Z2VyLmtlcm5lbC5vcmcgTW9yZQ0K Pj4gbWFqb3Jkb21vDQo+PiA+PiBpbmZvIGF0ICBodHRwOi8vdmdlci5rZXJuZWwub3JnL21ham9y ZG9tby1pbmZvLmh0bWwNCj4+IC0tDQo+PiBUbyB1bnN1YnNjcmliZSBmcm9tIHRoaXMgbGlzdDog c2VuZCB0aGUgbGluZSAidW5zdWJzY3JpYmUgY2VwaC1kZXZlbCIgaW4gdGhlDQo+PiBib2R5IG9m IGEgbWVzc2FnZSB0byBtYWpvcmRvbW9Admdlci5rZXJuZWwub3JnIE1vcmUgbWFqb3Jkb21vIGlu Zm8gYXQNCj4+IGh0dHA6Ly92Z2VyLmtlcm5lbC5vcmcvbWFqb3Jkb21vLWluZm8uaHRtbA0K -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
DQoNCk9uIFdlZG5lc2RheSAwMSBKdW5lIDIwMTYgMDg6NTQgQU0sIEppYW5qaWFuIEh1byB3cm90 ZToNCj4gSGkgQWxsZW4sDQo+DQo+IE9uIFR1ZSwgTWF5IDMxLCAyMDE2IGF0IDc6NTIgUE0sIEFs bGVuIFNhbXVlbHMgPEFsbGVuLlNhbXVlbHNAc2FuZGlzay5jb20+IHdyb3RlOg0KPj4+IC0tLS0t T3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+Pj4gRnJvbTogY2VwaC1kZXZlbC1vd25lckB2Z2VyLmtl cm5lbC5vcmcgW21haWx0bzpjZXBoLWRldmVsLQ0KPj4+IG93bmVyQHZnZXIua2VybmVsLm9yZ10g T24gQmVoYWxmIE9mIEhhb21haSBXYW5nDQo+Pj4gU2VudDogVHVlc2RheSwgTWF5IDMxLCAyMDE2 IDc6MzggUE0NCj4+PiBUbzogSmlhbmppYW4gSHVvIDxqaWFuamlhbi5odW9Ac2Ftc3VuZy5jb20+ DQo+Pj4gQ2M6IFNhZ2UgV2VpbCA8c3dlaWxAcmVkaGF0LmNvbT47IE1hcmsgTmVsc29uIDxtbmVs c29uQHJlZGhhdC5jb20+Ow0KPj4+IGNlcGgtZGV2ZWxAdmdlci5rZXJuZWwub3JnDQo+Pj4gU3Vi amVjdDogUmU6IFJvY2tzREIgSW5jb3JyZWN0IEFQSSBVc2FnZQ0KPj4+DQo+Pj4gT24gV2VkLCBK dW4gMSwgMjAxNiBhdCAzOjQwIEFNLCBKaWFuamlhbiBIdW8gPGppYW5qaWFuLmh1b0BzYW1zdW5n LmNvbT4NCj4+PiB3cm90ZToNCj4+Pj4gT24gVHVlLCBNYXkgMzEsIDIwMTYgYXQgMTE6MTIgQU0s IEhhb21haSBXYW5nIDxoYW9tYWlAeHNreS5jb20+DQo+Pj4gd3JvdGU6DQo+Pj4+PiBPbiBXZWQs IEp1biAxLCAyMDE2IGF0IDI6MDggQU0sIEppYW5qaWFuIEh1bw0KPj4+IDxqaWFuamlhbi5odW9A c2Ftc3VuZy5jb20+IHdyb3RlOg0KPj4+Pj4+IEhpIEhhb21haSwNCj4+Pj4+Pg0KPj4+Pj4+IEkg bm90aWNlZCB0aGlzIGFzIHdlbGwsIGFuZCBtYWRlIHNhbWUgY2hhbmdlcyB0byBSb2Nrc0RCU3Rv cmUgaW4gdGhpcyBQUg0KPj4+IGxhc3Qgd2VlazoNCj4+Pj4+PiBodHRwczovL2dpdGh1Yi5jb20v Y2VwaC9jZXBoL3B1bGwvOTIxNQ0KPj4+Pj4+DQo+Pj4+Pj4gT25lIHRoaW5nIHdoaWNoIGlzIGV2 ZW4gd29yc2UsICBzZWVrIHdpbGwgYnlwYXNzIHJvdyBjYWNoZSwgc28ga3YgcGFpcnMNCj4+PiB3 b24ndCBiZSBhYmxlIHRvIGJlIGNhY2hlZCBpbiByb3cgY2FjaGUuDQo+Pj4+Pj4gSSBhbSB3b3Jr aW5nIHRvIGJlbmNobWFyayB0aGUgcGVyZm9ybWFuY2UgaW1wYWN0LCB3aWxsIHB1Ymxpc2ggdGhl DQo+Pj4gcmVzdWx0cyBhZnRlciBJIGFtIGRvbmUgdGhpcyB3ZWVrLg0KPj4+Pj4gT2gsIGNvb2wh IEkgdGhpbmsgeW91IGNhbiBjaGVycnktcGljayBteSBhbm90aGVyIGxldmVsZGIgZml4Lg0KPj4+ PiBObyBwcm9ibGVtLCBJIHdpbGwgZG8gdGhhdC4NCj4+Pj4+IEJUVywgZG8geW91IHBheSBhbiBh dHRlbnRpb24gdG8gcHJlZml4IHNlZWsgYXBpPyBJIHRoaW5rIGl0IHdpbGwgYmUNCj4+Pj4+IG1v cmUgc3VpdGFibGUgdGhhbiBjb2x1bW4gZmFtaWx5IGluIGNlcGggY2FzZS4gSWYgd2UgY2FuIGhh dmUNCj4+Pj4+IHdlbGwtZGVmaW5lZCBwcmVmaXggcnVsZSwgd2UgY2FuIG1ha2UgbW9zdCBvZiBy YW5nZSBxdWVyeSBjaGVhcGVyIQ0KPj4+PiBXaGVuIGtleXMgd2l0aCBzYW1lIHByZWZpeCBhcmUg c3RvcmVkIGluIHRoZWlyIG93biBTU1QgZmlsZXMoQ0YgY2FzZSksDQo+Pj4gZXZlbiBzZWVraW5n IHdpdGhvdXQgcHJlZml4IHdpbGwgYmUgZmFzdGVyIHRoYW4gc2Vla2luZyB3aXRoIHByZWZpeCBi dXQgbWl4ZWQNCj4+PiB3aXRoIGRpZmZlcmVudCBwcmVmaXhlZCBrZXlzPw0KPj4+DQo+Pj4gRnJv bSBteSBjdXJyZW50IHZpZXcsIHByZWZpeCBjb3VsZCBiZSBtb3JlIGZsZXhpYmxlLiBGb3IgZXhh bXBsZSwgZWFjaCByZ3cNCj4+PiBidWNrZXQgaW5kZXggY291bGQgdXNlIG9uZSBwcmVmaXggdG8g bWFrZSBlYWNoIGJ1Y2tldCBpbmRleCBvYmplY3Qgc2Vlaw0KPj4+IHNlcGFyYXRlZC4gRm9yIENG LCBpdCB3b3VsZCBiZSB0b28gaGVhdnJ5Lg0KPj4gVGhpcyBtaWdodCBjYXVzZSBvdGhlciBwcm9i bGVtcy4gV291bGQgaXQgZHJhbWF0aWNhbGx5IGluY3JlYXNlIHRoZSBudW1iZXIgb2YgZmlsZXMg dGhhdCBCbHVlRlMgbmVlZHMgdG8gbWFuYWdlPyBJZiBzbywgdGhhdCBtaWdodCBlZmZlY3RpdmVs eSBicmVhayB0aGF0IGNvZGUgdG9vIChvZiBjb3Vyc2UgaXQncyBmaXhhYmxlIGFsc28gOikpDQo+ IFRoYW5rcyBmb3IgYnJpbmdpbmcgdXAgdGhpcyBpc3N1ZS4gV2hhdCdzIGN1cnJlbnQgbGltaXQg b2YgbnVtYmVyIG9mIGZpbGVzIGZvciBCbHVlRlMsIGZyb20geW91ciBleHBlcmllbmNlPw0KPiBK aWFuamlhbg0KVGhlcmUgaXMgbm8gbGltaXQgcmlnaHQgbm93LCBmbm9kZSBpbm8gaXMgYSB1bml0 NjRfdC4gIERpcmVjdG9yeSBpcyBhDQpyZWYgY291bnRlZCBvYmplY3QgY29udGFpbnMgYSBmaWxl IG1hcC4gQXMgdGhlIG51bWJlciBvZiBpbm9kZXMgZ3JvdywNCmZpbGVfbWFwIHdpbGwgZ3Jvdywg d2lsbCBjb25zdW1lIG1vcmUgbWVtb3J5Lg0KDQpWYXJhZGENCj4+Pj4gSSBhbSBub3Qgc3VyZSB3 aGF0IG9wdGltaXphdGlvbiBwcmVmaXggc2VlayB3aWxsIHVzZSBpbnRlcm5hbGx5IGZvciBibG9j aw0KPj4+IGJhc2VkIGZvcm1hdCwgYnV0IHRvIG1lLCBpdCdzIGhhcmQgdG8gYmVhdCB0aGUgY2Fz ZSB3aGVuIHlvdSBvbmx5IGhhdmUgb25lDQo+Pj4gcHJlZml4ZWQga2V5cyBzdG9yZWQgc2VwYXJh dGVseS4NCj4+Pj4+PiBKaWFuamlhbg0KPj4+Pj4+DQo+Pj4+Pj4gT24gVHVlLCBNYXkgMzEsIDIw MTYgYXQgOTo0OSBBTSwgSGFvbWFpIFdhbmcgPGhhb21haUB4c2t5LmNvbT4NCj4+PiB3cm90ZToN Cj4+Pj4+Pj4gSGkgU2FnZSBhbmQgTWFyaywNCj4+Pj4+Pj4NCj4+Pj4+Pj4gQXMgbWVudGlvbmVk IGluIEJsdWVTdG9yZSBzdGFuZHVwLCBJIGZvdW5kIHJvY2tzZGIgaXRlcmF0b3IgKlNlZWsqDQo+ Pj4+Pj4+IHdvbid0IHVzZSBibG9vbSBmaWx0ZXIgbGlrZSAqR2V0Ki4NCj4+Pj4+Pj4NCj4+Pj4+ Pj4gKkdldCogaW1wbDogaXQgd2lsbCBsb29rIGF0IGZpbHRlciBmaXJzdGx5DQo+Pj4+Pj4+DQo+ Pj4gaHR0cHM6Ly9naXRodWIuY29tL2ZhY2Vib29rL3JvY2tzZGIvYmxvYi9tYXN0ZXIvdGFibGUv YmxvY2tfYmFzZWRfdA0KPj4+Pj4+PiBhYmxlX3JlYWRlci5jYyNMMTM2OQ0KPj4+Pj4+Pg0KPj4+ Pj4+PiBJdGVyYXRvciAqU2Vlayo6IGl0IHdpbGwgZG8gYmluYXJ5IHNlYXJjaCwgYnkgZGVmYXVs dCB3ZSBkb24ndA0KPj4+Pj4+PiBzcGVjaWZ5IHByZWZpeA0KPj4+IGZlYXR1cmUoaHR0cHM6Ly9n aXRodWIuY29tL2ZhY2Vib29rL3JvY2tzZGIvd2lraS9QcmVmaXgtU2Vlay1BUEktDQo+Pj4gQ2hh bmdlcykuDQo+Pj4+Pj4+IGh0dHBzOi8vZ2l0aHViLmNvbS9mYWNlYm9vay9yb2Nrc2RiL2Jsb2Iv bWFzdGVyL3RhYmxlL2Jsb2NrLmNjI0w5NA0KPj4+Pj4+Pg0KPj4+Pj4+PiBTbyBJIHVzZSBhIHNp bXBsZSB0ZXN0czoNCj4+Pj4+Pj4NCj4+Pj4+Pj4gLi9kYl9iZW5jaCAtbnVtIDEwMDAwMDAwICAt YmVuY2htYXJrcyBmaWxsYmF0Y2ggZmlsbCB0aGUgZGIgZmlyc3RseQ0KPj4+Pj4+PiB3aXRoIDEw MDB3IHJlY29yZHMuDQo+Pj4+Pj4+DQo+Pj4+Pj4+IC4vZGJfYmVuY2ggLXVzZV9leGlzdGluZ19k YiAgLWJlbmNobWFya3MgcmVhZHJhbmRvbWZhc3QNCj4+Pj4+Pj4gcmVhZHJhbmRvbWZhc3QgY2Fz ZSB3aWxsIHVzZSAqR2V0KiBBUEkgdG8gcmV0cml2ZSBkYXRhDQo+Pj4+Pj4+DQo+Pj4+Pj4+IFty b290QGh1bnRlci1ub2RlMiByb2Nrc2RiXSMgLi9kYl9iZW5jaCAtdXNlX2V4aXN0aW5nX2RiDQo+ Pj4+Pj4+IC1iZW5jaG1hcmtzIHJlYWRyYW5kb21mYXN0DQo+Pj4+Pj4+DQo+Pj4+Pj4+IExldmVs REI6ICAgIHZlcnNpb24gNC4zDQo+Pj4+Pj4+IERhdGU6ICAgICAgIFdlZCBKdW4gIDEgMDA6Mjk6 MTYgMjAxNg0KPj4+Pj4+PiBDUFU6ICAgICAgICAzMiAqIEludGVsKFIpIFhlb24oUikgQ1BVIEU1 LTI2ODAgMCBAIDIuNzBHSHoNCj4+Pj4+Pj4gQ1BVQ2FjaGU6ICAgMjA0ODAgS0INCj4+Pj4+Pj4g S2V5czogICAgICAgMTYgYnl0ZXMgZWFjaA0KPj4+Pj4+PiBWYWx1ZXM6ICAgICAxMDAgYnl0ZXMg ZWFjaCAoNTAgYnl0ZXMgYWZ0ZXIgY29tcHJlc3Npb24pDQo+Pj4+Pj4+IEVudHJpZXM6ICAgIDEw MDAwMDANCj4+Pj4+Pj4gUHJlZml4OiAgICAwIGJ5dGVzDQo+Pj4+Pj4+IEtleXMgcGVyIHByZWZp eDogICAgMA0KPj4+Pj4+PiBSYXdTaXplOiAgICAxMTAuNiBNQiAoZXN0aW1hdGVkKQ0KPj4+Pj4+ PiBGaWxlU2l6ZTogICA2Mi45IE1CIChlc3RpbWF0ZWQpDQo+Pj4+Pj4+IFdyaXRlcyBwZXIgc2Vj b25kOiAwDQo+Pj4+Pj4+IENvbXByZXNzaW9uOiBTbmFwcHkNCj4+Pj4+Pj4gTWVtdGFibGVyZXA6 IHNraXBfbGlzdA0KPj4+Pj4+PiBQZXJmIExldmVsOiAwDQo+Pj4+Pj4+IFdBUk5JTkc6IEFzc2Vy dGlvbnMgYXJlIGVuYWJsZWQ7IGJlbmNobWFya3MgdW5uZWNlc3NhcmlseSBzbG93DQo+Pj4+Pj4+ IC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPj4+Pj4+ PiBEQiBwYXRoOiBbL3RtcC9yb2Nrc2RidGVzdC0wL2RiYmVuY2hdDQo+Pj4+Pj4+IHJlYWRyYW5k b21mYXN0IDogICAgICAgNC41NzAgbWljcm9zL29wIDIxODgwNiBvcHMvc2VjOyAoMTAwMDEwMCBv Zg0KPj4+Pj4+PiAxMDAwMTAwIGZvdW5kLCBpc3N1ZWQgNDY2Mzkgbm9uLWV4aXN0IGtleXMpDQo+ Pj4+Pj4+DQo+Pj4+Pj4+ID09PT09PT09PT09PT09PT09PT09PT09PT09PQ0KPj4+Pj4+PiB0aGVu IEkgbW9kaWZ5IHJlYWRyYW5kb21mYXN0IHRvIHVzZSBJdGVyYXRvciBBUElbMF06DQo+Pj4+Pj4+ DQo+Pj4+Pj4+IFtyb290QGh1bnRlci1ub2RlMiByb2Nrc2RiXSMgLi9kYl9iZW5jaCAtdXNlX2V4 aXN0aW5nX2RiDQo+Pj4+Pj4+IC1iZW5jaG1hcmtzIHJlYWRyYW5kb21mYXN0DQo+Pj4+Pj4+IExl dmVsREI6ICAgIHZlcnNpb24gNC4zDQo+Pj4+Pj4+IERhdGU6ICAgICAgIFdlZCBKdW4gIDEgMDA6 MzM6MDMgMjAxNg0KPj4+Pj4+PiBDUFU6ICAgICAgICAzMiAqIEludGVsKFIpIFhlb24oUikgQ1BV IEU1LTI2ODAgMCBAIDIuNzBHSHoNCj4+Pj4+Pj4gQ1BVQ2FjaGU6ICAgMjA0ODAgS0INCj4+Pj4+ Pj4gS2V5czogICAgICAgMTYgYnl0ZXMgZWFjaA0KPj4+Pj4+PiBWYWx1ZXM6ICAgICAxMDAgYnl0 ZXMgZWFjaCAoNTAgYnl0ZXMgYWZ0ZXIgY29tcHJlc3Npb24pDQo+Pj4+Pj4+IEVudHJpZXM6ICAg IDEwMDAwMDANCj4+Pj4+Pj4gUHJlZml4OiAgICAwIGJ5dGVzDQo+Pj4+Pj4+IEtleXMgcGVyIHBy ZWZpeDogICAgMA0KPj4+Pj4+PiBSYXdTaXplOiAgICAxMTAuNiBNQiAoZXN0aW1hdGVkKQ0KPj4+ Pj4+PiBGaWxlU2l6ZTogICA2Mi45IE1CIChlc3RpbWF0ZWQpDQo+Pj4+Pj4+IFdyaXRlcyBwZXIg c2Vjb25kOiAwDQo+Pj4+Pj4+IENvbXByZXNzaW9uOiBTbmFwcHkNCj4+Pj4+Pj4gTWVtdGFibGVy ZXA6IHNraXBfbGlzdA0KPj4+Pj4+PiBQZXJmIExldmVsOiAwDQo+Pj4+Pj4+IFdBUk5JTkc6IEFz c2VydGlvbnMgYXJlIGVuYWJsZWQ7IGJlbmNobWFya3MgdW5uZWNlc3NhcmlseSBzbG93DQo+Pj4+ Pj4+IC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPj4+ Pj4+PiBEQiBwYXRoOiBbL3RtcC9yb2Nrc2RidGVzdC0wL2RiYmVuY2hdDQo+Pj4+Pj4+IHJlYWRy YW5kb21mYXN0IDogICAgICA0NS4xODggbWljcm9zL29wIDIyMTI5IG9wcy9zZWM7ICgxMDAwMTAw IG9mDQo+Pj4+Pj4+IDEwMDAxMDAgZm91bmQsIGlzc3VlZCA0NjYzOSBub24tZXhpc3Qga2V5cykN Cj4+Pj4+Pj4NCj4+Pj4+Pj4NCj4+Pj4+Pj4gNDUuMTggdXMvb3AgdnMgNC41N3VzL29wIQ0KPj4+ Pj4+Pg0KPj4+Pj4+PiBUaGUgdGVzdCBjYW4gYmUgcmVwZWF0ZWQgYW5kIGVhc3kgdG8gZG8hIFBs eiBjb3JyZWN0IGlmIEknbSBkb2luZw0KPj4+Pj4+PiBmb29saXNoIHRoaW5nIEknbSBub3QgYXdh cmUuLg0KPj4+Pj4+Pg0KPj4+Pj4+PiBTbyBJIHByb3Bvc2FsIHRoaXMgUFI6IGh0dHBzOi8vZ2l0 aHViLmNvbS9jZXBoL2NlcGgvcHVsbC85NDExDQo+Pj4+Pj4+DQo+Pj4+Pj4+IFdlIHN0aWxsIGNh biBtYWtlIGZ1cnRoZXIgaW1wcm92ZW1lbnRzIGJ5IHNjYW5uaW5nIGFsbCBpdGVyYXRlDQo+Pj4+ Pj4+IHVzYWdlIHRvIG1ha2UgaXQgYmV0dGVyIQ0KPj4+Pj4+Pg0KPj4+Pj4+PiBbMF06DQo+Pj4+ Pj4+IC0tLSBhL2RiL2RiX2JlbmNoLmNjDQo+Pj4+Pj4+ICsrKyBiL2RiL2RiX2JlbmNoLmNjDQo+ Pj4+Pj4+IEBAIC0yOTIzLDE0ICsyOTIzLDEyIEBAIGNsYXNzIEJlbmNobWFyayB7DQo+Pj4+Pj4+ ICAgICAgICAgIGludDY0X3Qga2V5X3JhbmQgPSB0aHJlYWQtPnJhbmQuTmV4dCgpICYgKHBvdCAt IDEpOw0KPj4+Pj4+PiAgICAgICAgICBHZW5lcmF0ZUtleUZyb21JbnQoa2V5X3JhbmQsIEZMQUdT X251bSwgJmtleSk7DQo+Pj4+Pj4+ICAgICAgICAgICsrcmVhZDsNCj4+Pj4+Pj4gLSAgICAgICAg YXV0byBzdGF0dXMgPSBkYi0+R2V0KG9wdGlvbnMsIGtleSwgJnZhbHVlKTsNCj4+Pj4+Pj4gLSAg ICAgICAgaWYgKHN0YXR1cy5vaygpKSB7DQo+Pj4+Pj4+IC0gICAgICAgICAgKytmb3VuZDsNCj4+ Pj4+Pj4gLSAgICAgICAgfSBlbHNlIGlmICghc3RhdHVzLklzTm90Rm91bmQoKSkgew0KPj4+Pj4+ PiAtICAgICAgICAgIGZwcmludGYoc3RkZXJyLCAiR2V0IHJldHVybmVkIGFuIGVycm9yOiAlc1xu IiwNCj4+Pj4+Pj4gLSAgICAgICAgICAgICAgICAgIHN0YXR1cy5Ub1N0cmluZygpLmNfc3RyKCkp Ow0KPj4+Pj4+PiAtICAgICAgICAgIGFib3J0KCk7DQo+Pj4+Pj4+IC0gICAgICAgIH0NCj4+Pj4+ Pj4gKyAgICAgICAgSXRlcmF0b3IqIGl0ZXIgPSBkYi0+TmV3SXRlcmF0b3Iob3B0aW9ucyk7DQo+ Pj4+Pj4+ICsgICAgICBpdGVyLT5TZWVrKGtleSk7DQo+Pj4+Pj4+ICsgICAgICBpZiAoaXRlci0+ VmFsaWQoKSAmJiBpdGVyLT5rZXkoKS5jb21wYXJlKGtleSkgPT0gMCkgew0KPj4+Pj4+PiArICAg ICAgICBmb3VuZCsrOw0KPj4+Pj4+PiArICAgICAgfQ0KPj4+Pj4+PiArDQo+Pj4+Pj4+ICAgICAg ICAgIGlmIChrZXlfcmFuZCA+PSBGTEFHU19udW0pIHsNCj4+Pj4+Pj4gICAgICAgICAgICArK25v bmV4aXN0Ow0KPj4+Pj4+PiAgICAgICAgICB9DQo+Pj4+Pj4+IC0tDQo+Pj4+Pj4+IFRvIHVuc3Vi c2NyaWJlIGZyb20gdGhpcyBsaXN0OiBzZW5kIHRoZSBsaW5lICJ1bnN1YnNjcmliZQ0KPj4+Pj4+ PiBjZXBoLWRldmVsIiBpbiB0aGUgYm9keSBvZiBhIG1lc3NhZ2UgdG8gbWFqb3Jkb21vQHZnZXIu a2VybmVsLm9yZw0KPj4+Pj4+PiBNb3JlIG1ham9yZG9tbyBpbmZvIGF0ICBodHRwOi8vdmdlci5r ZXJuZWwub3JnL21ham9yZG9tby1pbmZvLmh0bWwNCj4+Pj4+IC0tDQo+Pj4+PiBUbyB1bnN1YnNj cmliZSBmcm9tIHRoaXMgbGlzdDogc2VuZCB0aGUgbGluZSAidW5zdWJzY3JpYmUgY2VwaC1kZXZl bCINCj4+Pj4+IGluIHRoZSBib2R5IG9mIGEgbWVzc2FnZSB0byBtYWpvcmRvbW9Admdlci5rZXJu ZWwub3JnIE1vcmUNCj4+PiBtYWpvcmRvbW8NCj4+Pj4+IGluZm8gYXQgIGh0dHA6Ly92Z2VyLmtl cm5lbC5vcmcvbWFqb3Jkb21vLWluZm8uaHRtbA0KPj4+IC0tDQo+Pj4gVG8gdW5zdWJzY3JpYmUg ZnJvbSB0aGlzIGxpc3Q6IHNlbmQgdGhlIGxpbmUgInVuc3Vic2NyaWJlIGNlcGgtZGV2ZWwiIGlu IHRoZQ0KPj4+IGJvZHkgb2YgYSBtZXNzYWdlIHRvIG1ham9yZG9tb0B2Z2VyLmtlcm5lbC5vcmcg TW9yZSBtYWpvcmRvbW8gaW5mbyBhdA0KPj4+IGh0dHA6Ly92Z2VyLmtlcm5lbC5vcmcvbWFqb3Jk b21vLWluZm8uaHRtbA0KPiBO77+977+977+977+977+9cu+/ve+/vXnvv73vv73vv71i77+9WO+/ ve+/vcendu+/vV7vv70p3rp7Lm7vv70r77+977+977+9eu+/vV1677+977+977+9e2F577+9yofa me+/vSxq77+977+9Zu+/ve+/ve+/vWjvv73vv73vv71677+977+9d++/ve+/ve+/ve+/ve+/ve+/ vWo6K3bvv73vv73vv71377+9au+/vW3vv73vv73vv73vv73vv73vv73vv73vv716Wivvv73vv73v v73vv73vv73domoi77+977+9Ie+/vWkNCg0KUExFQVNFIE5PVEU6IFRoZSBpbmZvcm1hdGlvbiBj b250YWluZWQgaW4gdGhpcyBlbGVjdHJvbmljIG1haWwgbWVzc2FnZSBpcyBpbnRlbmRlZCBvbmx5 IGZvciB0aGUgdXNlIG9mIHRoZSBkZXNpZ25hdGVkIHJlY2lwaWVudChzKSBuYW1lZCBhYm92ZS4g SWYgdGhlIHJlYWRlciBvZiB0aGlzIG1lc3NhZ2UgaXMgbm90IHRoZSBpbnRlbmRlZCByZWNpcGll bnQsIHlvdSBhcmUgaGVyZWJ5IG5vdGlmaWVkIHRoYXQgeW91IGhhdmUgcmVjZWl2ZWQgdGhpcyBt ZXNzYWdlIGluIGVycm9yIGFuZCB0aGF0IGFueSByZXZpZXcsIGRpc3NlbWluYXRpb24sIGRpc3Ry aWJ1dGlvbiwgb3IgY29weWluZyBvZiB0aGlzIG1lc3NhZ2UgaXMgc3RyaWN0bHkgcHJvaGliaXRl ZC4gSWYgeW91IGhhdmUgcmVjZWl2ZWQgdGhpcyBjb21tdW5pY2F0aW9uIGluIGVycm9yLCBwbGVh c2Ugbm90aWZ5IHRoZSBzZW5kZXIgYnkgdGVsZXBob25lIG9yIGUtbWFpbCAoYXMgc2hvd24gYWJv dmUpIGltbWVkaWF0ZWx5IGFuZCBkZXN0cm95IGFueSBhbmQgYWxsIGNvcGllcyBvZiB0aGlzIG1l c3NhZ2UgaW4geW91ciBwb3NzZXNzaW9uICh3aGV0aGVyIGhhcmQgY29waWVzIG9yIGVsZWN0cm9u aWNhbGx5IHN0b3JlZCBjb3BpZXMpLg0K -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
> -----Original Message----- > From: Varada Kari > Sent: Tuesday, May 31, 2016 8:38 PM > To: Jianjian Huo <jianjian.huo@samsung.com>; Allen Samuels > <Allen.Samuels@sandisk.com>; Haomai Wang <haomai@xsky.com> > Cc: Sage Weil <sweil@redhat.com>; Mark Nelson <mnelson@redhat.com>; > ceph-devel@vger.kernel.org > Subject: Re: RocksDB Incorrect API Usage > > > > On Wednesday 01 June 2016 08:54 AM, Jianjian Huo wrote: > > Hi Allen, > > > > On Tue, May 31, 2016 at 7:52 PM, Allen Samuels > <Allen.Samuels@sandisk.com> wrote: > >>> -----Original Message----- > >>> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel- > >>> owner@vger.kernel.org] On Behalf Of Haomai Wang > >>> Sent: Tuesday, May 31, 2016 7:38 PM > >>> To: Jianjian Huo <jianjian.huo@samsung.com> > >>> Cc: Sage Weil <sweil@redhat.com>; Mark Nelson > <mnelson@redhat.com>; > >>> ceph-devel@vger.kernel.org > >>> Subject: Re: RocksDB Incorrect API Usage > >>> > >>> On Wed, Jun 1, 2016 at 3:40 AM, Jianjian Huo > >>> <jianjian.huo@samsung.com> > >>> wrote: > >>>> On Tue, May 31, 2016 at 11:12 AM, Haomai Wang <haomai@xsky.com> > >>> wrote: > >>>>> On Wed, Jun 1, 2016 at 2:08 AM, Jianjian Huo > >>> <jianjian.huo@samsung.com> wrote: > >>>>>> Hi Haomai, > >>>>>> > >>>>>> I noticed this as well, and made same changes to RocksDBStore in > >>>>>> this PR > >>> last week: > >>>>>> https://github.com/ceph/ceph/pull/9215 > >>>>>> > >>>>>> One thing which is even worse, seek will bypass row cache, so kv > >>>>>> pairs > >>> won't be able to be cached in row cache. > >>>>>> I am working to benchmark the performance impact, will publish > >>>>>> the > >>> results after I am done this week. > >>>>> Oh, cool! I think you can cherry-pick my another leveldb fix. > >>>> No problem, I will do that. > >>>>> BTW, do you pay an attention to prefix seek api? I think it will > >>>>> be more suitable than column family in ceph case. If we can have > >>>>> well-defined prefix rule, we can make most of range query cheaper! > >>>> When keys with same prefix are stored in their own SST files(CF > >>>> case), > >>> even seeking without prefix will be faster than seeking with prefix > >>> but mixed with different prefixed keys? > >>> > >>> From my current view, prefix could be more flexible. For example, > >>> each rgw bucket index could use one prefix to make each bucket index > >>> object seek separated. For CF, it would be too heavry. > >> This might cause other problems. Would it dramatically increase the > >> number of files that BlueFS needs to manage? If so, that might > >> effectively break that code too (of course it's fixable also :)) > > Thanks for bringing up this issue. What's current limit of number of files for > BlueFS, from your experience? > > Jianjian > There is no limit right now, fnode ino is a unit64_t. Directory is a ref counted > object contains a file map. As the number of inodes grow, file_map will grow, > will consume more memory. > > Varada Technically correct, my concern is one of performance for BlueFS journal compaction. > >>>> I am not sure what optimization prefix seek will use internally for > >>>> block > >>> based format, but to me, it's hard to beat the case when you only > >>> have one prefixed keys stored separately. > >>>>>> Jianjian > >>>>>> > >>>>>> On Tue, May 31, 2016 at 9:49 AM, Haomai Wang > <haomai@xsky.com> > >>> wrote: > >>>>>>> Hi Sage and Mark, > >>>>>>> > >>>>>>> As mentioned in BlueStore standup, I found rocksdb iterator > >>>>>>> *Seek* won't use bloom filter like *Get*. > >>>>>>> > >>>>>>> *Get* impl: it will look at filter firstly > >>>>>>> > >>> > https://github.com/facebook/rocksdb/blob/master/table/block_based_t > >>>>>>> able_reader.cc#L1369 > >>>>>>> > >>>>>>> Iterator *Seek*: it will do binary search, by default we don't > >>>>>>> specify prefix > >>> feature(https://github.com/facebook/rocksdb/wiki/Prefix-Seek-API- > >>> Changes). > >>>>>>> > https://github.com/facebook/rocksdb/blob/master/table/block.cc#L > >>>>>>> 94 > >>>>>>> > >>>>>>> So I use a simple tests: > >>>>>>> > >>>>>>> ./db_bench -num 10000000 -benchmarks fillbatch fill the db > >>>>>>> firstly with 1000w records. > >>>>>>> > >>>>>>> ./db_bench -use_existing_db -benchmarks readrandomfast > >>>>>>> readrandomfast case will use *Get* API to retrive data > >>>>>>> > >>>>>>> [root@hunter-node2 rocksdb]# ./db_bench -use_existing_db > >>>>>>> -benchmarks readrandomfast > >>>>>>> > >>>>>>> LevelDB: version 4.3 > >>>>>>> Date: Wed Jun 1 00:29:16 2016 > >>>>>>> CPU: 32 * Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz > >>>>>>> CPUCache: 20480 KB > >>>>>>> Keys: 16 bytes each > >>>>>>> Values: 100 bytes each (50 bytes after compression) > >>>>>>> Entries: 1000000 > >>>>>>> Prefix: 0 bytes > >>>>>>> Keys per prefix: 0 > >>>>>>> RawSize: 110.6 MB (estimated) > >>>>>>> FileSize: 62.9 MB (estimated) > >>>>>>> Writes per second: 0 > >>>>>>> Compression: Snappy > >>>>>>> Memtablerep: skip_list > >>>>>>> Perf Level: 0 > >>>>>>> WARNING: Assertions are enabled; benchmarks unnecessarily slow > >>>>>>> ------------------------------------------------ > >>>>>>> DB path: [/tmp/rocksdbtest-0/dbbench] > >>>>>>> readrandomfast : 4.570 micros/op 218806 ops/sec; (1000100 of > >>>>>>> 1000100 found, issued 46639 non-exist keys) > >>>>>>> > >>>>>>> =========================== > >>>>>>> then I modify readrandomfast to use Iterator API[0]: > >>>>>>> > >>>>>>> [root@hunter-node2 rocksdb]# ./db_bench -use_existing_db > >>>>>>> -benchmarks readrandomfast > >>>>>>> LevelDB: version 4.3 > >>>>>>> Date: Wed Jun 1 00:33:03 2016 > >>>>>>> CPU: 32 * Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz > >>>>>>> CPUCache: 20480 KB > >>>>>>> Keys: 16 bytes each > >>>>>>> Values: 100 bytes each (50 bytes after compression) > >>>>>>> Entries: 1000000 > >>>>>>> Prefix: 0 bytes > >>>>>>> Keys per prefix: 0 > >>>>>>> RawSize: 110.6 MB (estimated) > >>>>>>> FileSize: 62.9 MB (estimated) > >>>>>>> Writes per second: 0 > >>>>>>> Compression: Snappy > >>>>>>> Memtablerep: skip_list > >>>>>>> Perf Level: 0 > >>>>>>> WARNING: Assertions are enabled; benchmarks unnecessarily slow > >>>>>>> ------------------------------------------------ > >>>>>>> DB path: [/tmp/rocksdbtest-0/dbbench] > >>>>>>> readrandomfast : 45.188 micros/op 22129 ops/sec; (1000100 of > >>>>>>> 1000100 found, issued 46639 non-exist keys) > >>>>>>> > >>>>>>> > >>>>>>> 45.18 us/op vs 4.57us/op! > >>>>>>> > >>>>>>> The test can be repeated and easy to do! Plz correct if I'm > >>>>>>> doing foolish thing I'm not aware.. > >>>>>>> > >>>>>>> So I proposal this PR: https://github.com/ceph/ceph/pull/9411 > >>>>>>> > >>>>>>> We still can make further improvements by scanning all iterate > >>>>>>> usage to make it better! > >>>>>>> > >>>>>>> [0]: > >>>>>>> --- a/db/db_bench.cc > >>>>>>> +++ b/db/db_bench.cc > >>>>>>> @@ -2923,14 +2923,12 @@ class Benchmark { > >>>>>>> int64_t key_rand = thread->rand.Next() & (pot - 1); > >>>>>>> GenerateKeyFromInt(key_rand, FLAGS_num, &key); > >>>>>>> ++read; > >>>>>>> - auto status = db->Get(options, key, &value); > >>>>>>> - if (status.ok()) { > >>>>>>> - ++found; > >>>>>>> - } else if (!status.IsNotFound()) { > >>>>>>> - fprintf(stderr, "Get returned an error: %s\n", > >>>>>>> - status.ToString().c_str()); > >>>>>>> - abort(); > >>>>>>> - } > >>>>>>> + Iterator* iter = db->NewIterator(options); > >>>>>>> + iter->Seek(key); > >>>>>>> + if (iter->Valid() && iter->key().compare(key) == 0) { > >>>>>>> + found++; > >>>>>>> + } > >>>>>>> + > >>>>>>> if (key_rand >= FLAGS_num) { > >>>>>>> ++nonexist; > >>>>>>> } > >>>>>>> -- > >>>>>>> To unsubscribe from this list: send the line "unsubscribe > >>>>>>> ceph-devel" in the body of a message to > >>>>>>> majordomo@vger.kernel.org More majordomo info at > >>>>>>> http://vger.kernel.org/majordomo-info.html > >>>>> -- > >>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" > >>>>> in the body of a message to majordomo@vger.kernel.org More > >>> majordomo > >>>>> info at http://vger.kernel.org/majordomo-info.html > >>> -- > >>> To unsubscribe from this list: send the line "unsubscribe > >>> ceph-devel" in the body of a message to majordomo@vger.kernel.org > >>> More majordomo info at http://vger.kernel.org/majordomo-info.html > > N r y b X ǧv ^ ){.n + z ]z {ay ʇڙ ,j f h z w j > > :+v w j m zZ+ ݢj" ! i
On Wednesday 01 June 2016 07:16 PM, Allen Samuels wrote: >> -----Original Message----- >> From: Varada Kari >> Sent: Tuesday, May 31, 2016 8:38 PM >> To: Jianjian Huo <jianjian.huo@samsung.com>; Allen Samuels >> <Allen.Samuels@sandisk.com>; Haomai Wang <haomai@xsky.com> >> Cc: Sage Weil <sweil@redhat.com>; Mark Nelson <mnelson@redhat.com>; >> ceph-devel@vger.kernel.org >> Subject: Re: RocksDB Incorrect API Usage >> >> >> >> On Wednesday 01 June 2016 08:54 AM, Jianjian Huo wrote: >>> Hi Allen, >>> >>> On Tue, May 31, 2016 at 7:52 PM, Allen Samuels >> <Allen.Samuels@sandisk.com> wrote: >>>>> -----Original Message----- >>>>> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel- >>>>> owner@vger.kernel.org] On Behalf Of Haomai Wang >>>>> Sent: Tuesday, May 31, 2016 7:38 PM >>>>> To: Jianjian Huo <jianjian.huo@samsung.com> >>>>> Cc: Sage Weil <sweil@redhat.com>; Mark Nelson >> <mnelson@redhat.com>; >>>>> ceph-devel@vger.kernel.org >>>>> Subject: Re: RocksDB Incorrect API Usage >>>>> >>>>> On Wed, Jun 1, 2016 at 3:40 AM, Jianjian Huo >>>>> <jianjian.huo@samsung.com> >>>>> wrote: >>>>>> On Tue, May 31, 2016 at 11:12 AM, Haomai Wang <haomai@xsky.com> >>>>> wrote: >>>>>>> On Wed, Jun 1, 2016 at 2:08 AM, Jianjian Huo >>>>> <jianjian.huo@samsung.com> wrote: >>>>>>>> Hi Haomai, >>>>>>>> >>>>>>>> I noticed this as well, and made same changes to RocksDBStore in >>>>>>>> this PR >>>>> last week: >>>>>>>> https://github.com/ceph/ceph/pull/9215 >>>>>>>> >>>>>>>> One thing which is even worse, seek will bypass row cache, so kv >>>>>>>> pairs >>>>> won't be able to be cached in row cache. >>>>>>>> I am working to benchmark the performance impact, will publish >>>>>>>> the >>>>> results after I am done this week. >>>>>>> Oh, cool! I think you can cherry-pick my another leveldb fix. >>>>>> No problem, I will do that. >>>>>>> BTW, do you pay an attention to prefix seek api? I think it will >>>>>>> be more suitable than column family in ceph case. If we can have >>>>>>> well-defined prefix rule, we can make most of range query cheaper! >>>>>> When keys with same prefix are stored in their own SST files(CF >>>>>> case), >>>>> even seeking without prefix will be faster than seeking with prefix >>>>> but mixed with different prefixed keys? >>>>> >>>>> From my current view, prefix could be more flexible. For example, >>>>> each rgw bucket index could use one prefix to make each bucket index >>>>> object seek separated. For CF, it would be too heavry. >>>> This might cause other problems. Would it dramatically increase the >>>> number of files that BlueFS needs to manage? If so, that might >>>> effectively break that code too (of course it's fixable also :)) >>> Thanks for bringing up this issue. What's current limit of number of files for >> BlueFS, from your experience? >>> Jianjian >> There is no limit right now, fnode ino is a unit64_t. Directory is a ref counted >> object contains a file map. As the number of inodes grow, file_map will grow, >> will consume more memory. >> >> Varada > Technically correct, my concern is one of performance for BlueFS journal compaction. > Yes. I am working on optimizing the latency part during the compaction. Varada >>>>>> I am not sure what optimization prefix seek will use internally for >>>>>> block >>>>> based format, but to me, it's hard to beat the case when you only >>>>> have one prefixed keys stored separately. >>>>>>>> Jianjian >>>>>>>> >>>>>>>> On Tue, May 31, 2016 at 9:49 AM, Haomai Wang >> <haomai@xsky.com> >>>>> wrote: >>>>>>>>> Hi Sage and Mark, >>>>>>>>> >>>>>>>>> As mentioned in BlueStore standup, I found rocksdb iterator >>>>>>>>> *Seek* won't use bloom filter like *Get*. >>>>>>>>> >>>>>>>>> *Get* impl: it will look at filter firstly >>>>>>>>> >> https://github.com/facebook/rocksdb/blob/master/table/block_based_t >>>>>>>>> able_reader.cc#L1369 >>>>>>>>> >>>>>>>>> Iterator *Seek*: it will do binary search, by default we don't >>>>>>>>> specify prefix >>>>> feature(https://github.com/facebook/rocksdb/wiki/Prefix-Seek-API- >>>>> Changes). >> https://github.com/facebook/rocksdb/blob/master/table/block.cc#L >>>>>>>>> 94 >>>>>>>>> >>>>>>>>> So I use a simple tests: >>>>>>>>> >>>>>>>>> ./db_bench -num 10000000 -benchmarks fillbatch fill the db >>>>>>>>> firstly with 1000w records. >>>>>>>>> >>>>>>>>> ./db_bench -use_existing_db -benchmarks readrandomfast >>>>>>>>> readrandomfast case will use *Get* API to retrive data >>>>>>>>> >>>>>>>>> [root@hunter-node2 rocksdb]# ./db_bench -use_existing_db >>>>>>>>> -benchmarks readrandomfast >>>>>>>>> >>>>>>>>> LevelDB: version 4.3 >>>>>>>>> Date: Wed Jun 1 00:29:16 2016 >>>>>>>>> CPU: 32 * Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz >>>>>>>>> CPUCache: 20480 KB >>>>>>>>> Keys: 16 bytes each >>>>>>>>> Values: 100 bytes each (50 bytes after compression) >>>>>>>>> Entries: 1000000 >>>>>>>>> Prefix: 0 bytes >>>>>>>>> Keys per prefix: 0 >>>>>>>>> RawSize: 110.6 MB (estimated) >>>>>>>>> FileSize: 62.9 MB (estimated) >>>>>>>>> Writes per second: 0 >>>>>>>>> Compression: Snappy >>>>>>>>> Memtablerep: skip_list >>>>>>>>> Perf Level: 0 >>>>>>>>> WARNING: Assertions are enabled; benchmarks unnecessarily slow >>>>>>>>> ------------------------------------------------ >>>>>>>>> DB path: [/tmp/rocksdbtest-0/dbbench] >>>>>>>>> readrandomfast : 4.570 micros/op 218806 ops/sec; (1000100 of >>>>>>>>> 1000100 found, issued 46639 non-exist keys) >>>>>>>>> >>>>>>>>> =========================== >>>>>>>>> then I modify readrandomfast to use Iterator API[0]: >>>>>>>>> >>>>>>>>> [root@hunter-node2 rocksdb]# ./db_bench -use_existing_db >>>>>>>>> -benchmarks readrandomfast >>>>>>>>> LevelDB: version 4.3 >>>>>>>>> Date: Wed Jun 1 00:33:03 2016 >>>>>>>>> CPU: 32 * Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz >>>>>>>>> CPUCache: 20480 KB >>>>>>>>> Keys: 16 bytes each >>>>>>>>> Values: 100 bytes each (50 bytes after compression) >>>>>>>>> Entries: 1000000 >>>>>>>>> Prefix: 0 bytes >>>>>>>>> Keys per prefix: 0 >>>>>>>>> RawSize: 110.6 MB (estimated) >>>>>>>>> FileSize: 62.9 MB (estimated) >>>>>>>>> Writes per second: 0 >>>>>>>>> Compression: Snappy >>>>>>>>> Memtablerep: skip_list >>>>>>>>> Perf Level: 0 >>>>>>>>> WARNING: Assertions are enabled; benchmarks unnecessarily slow >>>>>>>>> ------------------------------------------------ >>>>>>>>> DB path: [/tmp/rocksdbtest-0/dbbench] >>>>>>>>> readrandomfast : 45.188 micros/op 22129 ops/sec; (1000100 of >>>>>>>>> 1000100 found, issued 46639 non-exist keys) >>>>>>>>> >>>>>>>>> >>>>>>>>> 45.18 us/op vs 4.57us/op! >>>>>>>>> >>>>>>>>> The test can be repeated and easy to do! Plz correct if I'm >>>>>>>>> doing foolish thing I'm not aware.. >>>>>>>>> >>>>>>>>> So I proposal this PR: https://github.com/ceph/ceph/pull/9411 >>>>>>>>> >>>>>>>>> We still can make further improvements by scanning all iterate >>>>>>>>> usage to make it better! >>>>>>>>> >>>>>>>>> [0]: >>>>>>>>> --- a/db/db_bench.cc >>>>>>>>> +++ b/db/db_bench.cc >>>>>>>>> @@ -2923,14 +2923,12 @@ class Benchmark { >>>>>>>>> int64_t key_rand = thread->rand.Next() & (pot - 1); >>>>>>>>> GenerateKeyFromInt(key_rand, FLAGS_num, &key); >>>>>>>>> ++read; >>>>>>>>> - auto status = db->Get(options, key, &value); >>>>>>>>> - if (status.ok()) { >>>>>>>>> - ++found; >>>>>>>>> - } else if (!status.IsNotFound()) { >>>>>>>>> - fprintf(stderr, "Get returned an error: %s\n", >>>>>>>>> - status.ToString().c_str()); >>>>>>>>> - abort(); >>>>>>>>> - } >>>>>>>>> + Iterator* iter = db->NewIterator(options); >>>>>>>>> + iter->Seek(key); >>>>>>>>> + if (iter->Valid() && iter->key().compare(key) == 0) { >>>>>>>>> + found++; >>>>>>>>> + } >>>>>>>>> + >>>>>>>>> if (key_rand >= FLAGS_num) { >>>>>>>>> ++nonexist; >>>>>>>>> } >>>>>>>>> -- >>>>>>>>> To unsubscribe from this list: send the line "unsubscribe >>>>>>>>> ceph-devel" in the body of a message to >>>>>>>>> majordomo@vger.kernel.org More majordomo info at >>>>>>>>> http://vger.kernel.org/majordomo-info.html >>>>>>> -- >>>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" >>>>>>> in the body of a message to majordomo@vger.kernel.org More >>>>> majordomo >>>>>>> info at http://vger.kernel.org/majordomo-info.html >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe >>>>> ceph-devel" in the body of a message to majordomo@vger.kernel.org >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> N r y b X ǧv ^ ){.n + z ]z {ay ʇڙ ,j f h z w j >>> :+v w j m zZ+ ݢj" ! i > PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
=========================== then I modify readrandomfast to use Iterator API[0]: [root@hunter-node2 rocksdb]# ./db_bench -use_existing_db -benchmarks readrandomfast LevelDB: version 4.3 Date: Wed Jun 1 00:33:03 2016 CPU: 32 * Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz CPUCache: 20480 KB Keys: 16 bytes each Values: 100 bytes each (50 bytes after compression) Entries: 1000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 110.6 MB (estimated) FileSize: 62.9 MB (estimated) Writes per second: 0 Compression: Snappy Memtablerep: skip_list Perf Level: 0 WARNING: Assertions are enabled; benchmarks unnecessarily slow ------------------------------------------------ DB path: [/tmp/rocksdbtest-0/dbbench] readrandomfast : 45.188 micros/op 22129 ops/sec; (1000100 of 1000100 found, issued 46639 non-exist keys) 45.18 us/op vs 4.57us/op! The test can be repeated and easy to do! Plz correct if I'm doing foolish thing I'm not aware.. So I proposal this PR: https://github.com/ceph/ceph/pull/9411 We still can make further improvements by scanning all iterate usage to make it better! [0]: --- a/db/db_bench.cc +++ b/db/db_bench.cc @@ -2923,14 +2923,12 @@ class Benchmark { int64_t key_rand = thread->rand.Next() & (pot - 1); GenerateKeyFromInt(key_rand, FLAGS_num, &key); ++read; - auto status = db->Get(options, key, &value); - if (status.ok()) { - ++found; - } else if (!status.IsNotFound()) { - fprintf(stderr, "Get returned an error: %s\n", - status.ToString().c_str()); - abort(); - } + Iterator* iter = db->NewIterator(options); + iter->Seek(key); + if (iter->Valid() && iter->key().compare(key) == 0) { + found++; + } + if (key_rand >= FLAGS_num) { ++nonexist; }