added new cloud statistics facility. #25
Conversation
| delete it; | ||
|
|
||
| // Flush all data from main db to sst files. Release db. | ||
| db->Flush(FlushOptions()); |
There was a problem hiding this comment.
Can we print out the Statistics values into stdout? Then this example will be super clear that it is showing how Statistics work!
| #include "rocksdb/options.h" | ||
|
|
||
| using namespace rocksdb; | ||
|
|
There was a problem hiding this comment.
Don't you need to update examples/Makefile to compile and run this one?
| */ | ||
| enum CloudTickers : uint32_t { | ||
| // # of times the manifest is written to the cloud | ||
| CLOUD_TICKER_ENUM_START = 1u << 31, |
There was a problem hiding this comment.
Can we have some comments on why this needs to start at 1 << 31?
| @@ -464,6 +465,13 @@ Status S3WritableFile::CopyManifestToS3(uint64_t size_hint, bool force) { | |||
| "[s3] S3WritableFile made manifest %s durable to " | |||
| "bucket %s bucketpath %s.", | |||
| fname_.c_str(), s3_bucket_.c_str(), s3_object_.c_str()); | |||
|
|
|||
| // If cloud stats are present, record the manifest write and its latency in millis. | |||
| auto stats = env_->cloud_env_options.cloud_statistics; | |||
There was a problem hiding this comment.
Pls use auto&. Otherwise, you'll copy shared_ptr here, which needs to write to a std::atomic.
There was a problem hiding this comment.
On a related note, check out second paragraph here: http://smalldatum.blogspot.com/2016/10/make-myrocks-2x-less-slow.html
| @@ -244,6 +251,9 @@ class AwsEnv : public CloudEnvImpl { | |||
| // The S3 client | |||
| std::shared_ptr<AwsS3ClientWrapper> s3client_; | |||
|
|
|||
| // Results of last S3 client call; used in stats collection. | |||
| static thread_local AwsS3ClientResult s3client_result_; | |||
There was a problem hiding this comment.
Unfortunately we cannot use thread_local, since RocksDB can run on platforms that don't support it. We can use __thread, but only in some scenarios, check out how RocksDB does it: https://github.com/facebook/rocksdb/blob/master/monitoring/perf_context.cc#L20
| auto stats = env_->cloud_env_options.cloud_statistics; | ||
| if (stats) { | ||
| stats->recordTick(NUMBER_MANIFEST_WRITES, 1); | ||
| stats->measureTime(MANIFEST_WRITES_TIME, env_->s3client_result_.micros / 1000); |
There was a problem hiding this comment.
Could you just measure time here instead of piggy-backing on the time measure in S3 callback? That would make the code much easier to reason about (no need for thread-local, wrapping the callback, etc). Also, you already have the start time of the operation recorded in line 455, so adding an extra time call shouldn't be a big concern.
|
Hi @dlfurse, if you are ok with the review comments, please consider uploading a newer version of this patch. Then we can merge it into master. |
No description provided.