fix title bug and add English doc #100
Conversation
Co-authored-by: Haonan <[email protected]>
| In this example, two levels of storage are configured, specifically: | ||
|
|
||
| | **层级** | **数据目录** | **数据范围** | **磁盘最小剩余空间阈值** | |
There was a problem hiding this comment.
This part needs to be in English
| In this example, a total of three levels of storage are configured, specifically: | ||
|
|
||
| | **层级** | **数据目录** | **数据范围** | **磁盘最小剩余空间阈值** | |
There was a problem hiding this comment.
This part needs to be in English
| @@ -35,8 +33,4 @@ The research papers related are as follows: | |||
| * [The Design of Apache IoTDB distributed framework](http://ndbc2019.sdu.edu.cn/info/1002/1044.htm), Tianan Li, Jianmin Wang, Xiangdong Huang, Yi Xu, Dongfang Mao, Jun Yuan. NDBC 2019 | |||
| * [Dual-PISA: An index for aggregation operations on time series data](https://www.sciencedirect.com/science/article/pii/S0306437918305489), Jialin Qiao, Xiangdong Huang, Jianmin Wang, Raymond K Wong. IS 2020 | |||
|
|
|||
| ## Benchmark tools | |||
|
|
|||
| We also developed Benchmark tools for time series databases | |||
There was a problem hiding this comment.
Rather write: "benchmarking tools"
| Tiered storage function provides users with the ability to manage tiered storage media. users can use the tiered storage function to configure different types of storage media for IoTDB and to classify the storage media. ioTDB can support tiered storage from memory, SSD, normal hard disc to network hard disc by parameter configuration only according to the degree of hot and cold data. Specifically, in IoTDB, the configuration of tiered storage is reflected in the management of multiple directories. Users can group tiered storage directories into the same category and configure them into IoTDB as a "tier", which is called storage tier; at the same time, users can categorize data according to hot or cold, and store different categories of data into designated storage tiers. Meanwhile, users can categorise data according to hot or cold and store different categories of data in the specified tier. Currently, IoTDB supports the classification of hot and cold data by TTL, when the data in one tier does not meet the TTL rules defined in the current tier, the data will be automatically migrated to the next tier. |
There was a problem hiding this comment.
The Tiered storage functionality allows users to define multiple layers of storage, spanning across multiple types of storage media (Memory mapped directory, SSD, rotational hard discs or cloud storage). While memory and cloud storage is usually singular, the local file system storages can consist of multiple directories joined together into one tier. In order to decide where to store data, the probably most important configuration option is the TTL (Time to live / age) of data. Therefore for each tier a default TTL needs to be provided. Each tier also has a quota to how much data it can contain. When inserting data, IoTDB checks if the conditions for the first layer match (Is the data too old? Is the layer already saturated with too much data?) If any of the conditions don't match, the next tier is checked until a matching tier is found.
| | Configuration | Default | Description | Constraint | | ||
| | ---------------------------------------- | ------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | | ||
| | dn_data_dirs | None | specify different storage directories and divide the storage directories into tiers | Each level of storage uses a semicolon to separate, and commas to separate within a single level; cloud configuration can only be used as the last level of storage and the first level can't be used as cloud storage; a cloud object at most; the remote storage directory is denoted by OBJECT_STORAGE | |
There was a problem hiding this comment.
| dn_data_dirs | None | specify different storage directories and divide the storage directories into tiers | Each level of storage uses a semicolon to separate, and commas to separate within a single level; cloud (OBJECT_STORAGE) configuration can only be used as the last level of storage and the first level can't be used as cloud storage; a cloud object at most; the remote storage directory is denoted by OBJECT_STORAGE |
| | ---------------------------------------- | ------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | | ||
| | dn_data_dirs | None | specify different storage directories and divide the storage directories into tiers | Each level of storage uses a semicolon to separate, and commas to separate within a single level; cloud configuration can only be used as the last level of storage and the first level can't be used as cloud storage; a cloud object at most; the remote storage directory is denoted by OBJECT_STORAGE | | ||
| | default_ttl_in_ms | None | Define the scope of data for which each tier is responsible, expressed through a TTL | Each level of storage is separated by a semicolon; the number of levels should match the number of levels defined by dn_data_dirs | |
There was a problem hiding this comment.
| default_ttl_in_ms | None | Define the maximum age of data for which each tier is responsible | Each level of storage is separated by a semicolon; the number of levels should match the number of levels defined by dn_data_dirs |
| | object_storage_bucket | None | Name of cloud storage bucket | Bucket definition in AWS S3; no need to configure if remote storage is not used | | ||
| | object_storage_endpoiont | | endpoint of cloud storage | endpoint of AWS S3；If remote storage is not used, no configuration required | | ||
| | object_storage_access_key | | Authentication information stored in the cloud: key | AWS S3 的 credential key；If remote storage is not used, no configuration required | |
There was a problem hiding this comment.
| object_storage_access_key | | Authentication information stored in the cloud: key | AWS S3 credential key；If remote storage is not used, no configuration required |
| | object_storage_endpoiont | | endpoint of cloud storage | endpoint of AWS S3；If remote storage is not used, no configuration required | | ||
| | object_storage_access_key | | Authentication information stored in the cloud: key | AWS S3 的 credential key；If remote storage is not used, no configuration required | | ||
| | object_storage_access_secret | | Authentication information stored in the cloud: secret | AWS S3 的 credential secret；If remote storage is not used, no configuration required | |
There was a problem hiding this comment.
| object_storage_access_secret | | Authentication information stored in the cloud: secret | AWS S3 credential secret；If remote storage is not used, no configuration required |
| | -------- | -------------------------------------- | --------------- | ------------------------ | | ||
| | tier 1 | path 1：/data1/data | data for last 1 day | 20% | | ||
| | tier 2 | path 2：/data2/data path 2：/data3/data | data from 1 day ago | 10% | |
There was a problem hiding this comment.
| tier 2 | path 2：/data2/data path 2：/data3/data | all data | 10% |
| | **tier** | **data path** | **data range** | **threshold for minimum remaining disk space** | | ||
| | -------- | -------------------------------------- | ---------------------------- | ------------------------ | | ||
| | tier一 | path 1：/data1/data | data for last 1 day | 20% | |
There was a problem hiding this comment.
| tier | path 1：/data1/data | data for last 1 day | 20% |
| | -------- | -------------------------------------- | ---------------------------- | ------------------------ | | ||
| | tier一 | path 1：/data1/data | data for last 1 day | 20% | | ||
| | tier二 | path 1：/data2/data path 2：/data3/data | data from past 1 day to past 10 days | 15% | |
There was a problem hiding this comment.
| tier | path 1：/data2/data path 2：/data3/data | data from past 10 days | 15% |
| | tier一 | path 1：/data1/data | data for last 1 day | 20% | | ||
| | tier二 | path 1：/data2/data path 2：/data3/data | data from past 1 day to past 10 days | 15% | | ||
| | tier三 | Remote AWS S3 Storage | data from 1 day ago | 10% | |
There was a problem hiding this comment.
| tier | Remote AWS S3 Storage | all data | 10% |
No description provided.