Skip to content

Latest commit

 

History

History
169 lines (141 loc) · 8.73 KB

README.md

File metadata and controls

169 lines (141 loc) · 8.73 KB
Raw

Purpose of this module

opendb module provides observability and configurability for rocksdb databases.

Observability

When you open RocksDB with provided API:

  • opendb registers prometheus metrics
  • opendb launches a reportMetrics goroutine which every rocksdb.report-metrics-interval-secs seconds reports metrics to prometheus, reportMetrics goroutine performs following steps:

List of reported metrics and their documentation can be found in:

  • source code: registerMetrics() function in metrics.go
  • corresponding grafana dashboard

Configurability

opendb allows you to override default rocksdb settings, more info can be found here:

There are 2 main scenarios:

  • create a new rocksdb database
  • open already existing database

Create a new rocksdb database

Open an already existing database

  • opendb loads stored rocksdb configuration and starts with it
  • opendb overrides options which explicitly specified in appOpts (app.toml)

Individual database configuration

app.toml example:

[rocksdb]
enable-metrics = true
report-metrics-interval-secs = 15
max-open-files = 16384
...

[rocksdb.blockstore]
max-open-files = -1
block_size = 4096

[rocksdb.tx_index]
max-open-files = -1
block_size = 4096

[rocksdb] section contains default/fallback database configuration, but you can override it with database-specific configuration

it's very convenient approach, because rocksdb configuration is pretty extensive, contains around 30 parameters, but usually we override only few of them

so we can define standard configuration in [rocksdb] section, and then override only few params in database-specific configurations

List of databases:

Name Subsystem IAVL V1 size as of 10.5 millions blocks IAVL V1 number of SST files as of 10.5 millions blocks
application.db Kava 4.5 TB 76k
metadata.db (snapshots) Kava 21 MB 300
blockstore.db Ethermint 337 GB 6k
state.db Ethermint 282 GB 5.5k
tx_index.db Ethermint 504 GB 11.6k
evidence.db Ethermint 28 MB 270
evmindexer.db Ethermint 2.2 GB 230

List of reported rocksdb metrics:

Name Subsystem Docs
number_keys_written Key
number_keys_read Key
number_keys_updated Key
estimate_num_keys Key estimated number of total keys in the active and unflushed immutable memtables and storage
number_file_opens File
number_file_errors File
block_cache_usage Memory memory size for the entries residing in block cache
estimate_table_readers_mem Memory estimated memory used for reading SST tables, excluding memory used in block cache (e.g., filter and index blocks)
cur_size_all_mem_tables Memory approximate size of active and unflushed immutable memtables (bytes)
block_cache_pinned_usage Memory returns the memory size for the entries being pinned
block_cache_miss Cache block_cache_miss == block_cache_index_miss + block_cache_filter_miss + block_cache_data_miss
block_cache_hit Cache block_cache_hit == block_cache_index_hit + block_cache_filter_hit + block_cache_data_hit
block_cache_add Cache number of blocks added to block cache
block_cache_add_failures Cache number of failures when adding blocks to block cache
block_cache_index_miss Detailed Cache
block_cache_index_hit Detailed Cache
block_cache_index_bytes_insert Detailed Cache
block_cache_filter_miss Detailed Cache
block_cache_filter_hit Detailed Cache
block_cache_filter_bytes_insert Detailed Cache
block_cache_data_miss Detailed Cache
block_cache_data_hit Detailed Cache
block_cache_data_bytes_insert Detailed Cache
db_get_micros_p50 Latency
db_get_micros_p95 Latency
db_get_micros_p99 Latency
db_get_micros_p100 Latency
db_get_micros_count Latency
db_write_micros_p50 Latency
db_write_micros_p95 Latency
db_write_micros_p99 Latency
db_write_micros_p100 Latency
db_write_micros_count Latency
stall_micros Stall Writer has to wait for compaction or flush to finish.
db_write_stall_p50 Stall
db_write_stall_p95 Stall
db_write_stall_p99 Stall
db_write_stall_p100 Stall
db_write_stall_count Stall
db_write_stall_sum Stall
bloom_filter_useful Filter number of times bloom filter has avoided file reads, i.e., negatives.
bloom_filter_full_positive Filter number of times bloom FullFilter has not avoided the reads.
bloom_filter_full_true_positive Filter number of times bloom FullFilter has not avoided the reads and data actually exist.
last_level_read_bytes LSM
last_level_read_count LSM
non_last_level_read_bytes LSM
non_last_level_read_count LSM
get_hit_l0 LSM number of Get() queries served by L0
get_hit_l1 LSM number of Get() queries served by L1
get_hit_l2_and_up LSM number of Get() queries served by L2 and up

Example of RocksDB configuration

#####################
# RocksDB Tuning
#####################
rocksdb_enable_metrics: true
rocksdb_report_metrics_interval_secs: 15

rocksdb_max_open_files: 16384 # increase default max # of open files from 4096
rocksdb_max_file_opening_threads: 16
rocksdb_table_cache_numshardbits: 6
rocksdb_allow_mmap_writes: false
rocksdb_allow_mmap_reads: false
rocksdb_use_fsync: false
rocksdb_use_adaptive_mutex: false
rocksdb_bytes_per_sync: 0
rocksdb_max_background_jobs: 16

rocksdb_write_buffer_size: 134217728
rocksdb_num_levels: 7
rocksdb_max_write_buffer_number: 6
rocksdb_min_write_buffer_number_to_merge: 2
rocksdb_max_bytes_for_level_base: 536870912
rocksdb_max_bytes_for_level_multiplier: 10.000000
rocksdb_target_file_size_base: 67108864
rocksdb_target_file_size_multiplier: 1
rocksdb_level0_file_num_compaction_trigger: 2
rocksdb_level0_slowdown_writes_trigger: 20

rocksdb_block_cache_size: 1073741824
rocksdb_bits_per_key: 10
rocksdb_block_size: 16384 # 16K to match default zfs. Decreases block index memory usage by 4x from the default 4K
rocksdb_cache_index_and_filter_blocks: false
rocksdb_pin_l0_filter_and_index_blocks_in_cache: false
rocksdb_format_version: 5

# https://rocksdb.org/blog/2022/10/07/asynchronous-io-in-rocksdb.html
rocksdb_read_async_io: true # help speed up iterations