Skip to content

Commit

Permalink
add datasync and streaming doc for community version and fix wrong im…
Browse files Browse the repository at this point in the history
…g in benchmark doc (#143)
  • Loading branch information
wanghui42 authored Jan 10, 2024
1 parent 15631d8 commit 992aa87
Show file tree
Hide file tree
Showing 7 changed files with 2,538 additions and 63 deletions.
56 changes: 27 additions & 29 deletions src/UserGuide/V1.3.x/Tools-System/Benchmark.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,30 +25,28 @@ IoT-benchmark is a time-series database benchmarking tool based on Java and big

Figure 1-1 below includes the test benchmark process and other extended functions. These processes can be unified by IoT-benchmark. IoT Benchmark supports a variety of workloads, including **pure write, pure query, write query mixed**, etc., supports **software and hardware system monitoring, test metric measurement** and other monitoring functions, and also realizes **initializing the database automatically, test data analysis and system parameter optimization** functions.

![img](https://alioss.timecho.com/docs/img/1.PNG)
![img](https://alioss.timecho.com/docs/img/bm1.PNG)

Figure 1-1

Referring to the YCSB test tool's design idea of separating the three components of workload generation, performance metric measurement and database interface, the modular design of IoT-benchmark is shown in Figure 1-2. Different from the YCSB-based test tool system, IoT-benchmark adds a system monitoring module to support the persistence of test data and system monitoring data. In addition, some special load testing functions especially designed for time series data scenarios have been added, such as supporting batch writing and multiple out-of-sequence data writing modes for IoT scenarios.

![img](https://alioss.timecho.com/docs/img/2.PNG)
![img](https://alioss.timecho.com/docs/img/bm2.PNG)

Figure 1-2

Currently IoT-benchmark supports the following time series databases, versions and connection methods:

| Database | DB-Engines Rank | Version | Connection Methods |
| --------------- | --------------- | ------- | -------------------------------------------------------- |
| InfluxDB | 1 | v1.x | SDK |
| v2.0 | | | |
| TimescaleDB | 5 | -- | jdbc |
| OpenTSDB | 8 | -- | Http Request |
| QuestDB | 10 | v6.0.7 | jdbc |
| TDengine | 13 | 2.2.0.2 | jdbc |
| VictoriaMetrics | 16 | v1.64.0 | Http Request |
| KairosDB | 17 | -- | Http Request |
| IoTDB | 19 | v1.0 | jdbc、sessionByTablet、sessionByRecord、sessionByRecords |
| v0.13 | | | |
| Database | Version | Connection mmethod |
| --------------- | ------- | -------------------------------------------------------- |
| InfluxDB | v1.x <br> v2.0 | SDK | |
| TimescaleDB | -- | jdbc |
| OpenTSDB | -- | Http Request |
| QuestDB | v6.0.7 | jdbc |
| TDengine | v2.2.0.2 | jdbc |
| VictoriaMetrics | v1.64.0 | Http Request |
| KairosDB | -- | Http Request |
| IoTDB | v1.x <br> v0.13 | jdbc、sessionByTablet、sessionByRecord、sessionByRecords |

Table 1-1 Comparison of big data test benchmarks

Expand All @@ -71,7 +69,7 @@ Table 1-1 Comparison of big data test benchmarks

The directory structure of the test package is shown in Figure 1-3 below. The test configuration file is conf/config.properties, and the test startup scripts are benchmark.sh (Linux & MacOS) and benchmark.bat (Windows). The detailed usage of the files is shown in Table 1-2.

![img](https://alioss.timecho.com/docs/img/3.png)
![img](https://alioss.timecho.com/docs/img/bm3.png)

Figure 1-3 List of files and folders

Expand Down Expand Up @@ -99,7 +97,7 @@ Table 1-2 Usage list of files and folders

All the log files of the test are stored in the logs folder, and the test results are stored in the data/csvOutput folder after the test is completed. For example, after the test, we get the following result matrix:

![img](https://alioss.timecho.com/docs/img/4.png)
![img](https://alioss.timecho.com/docs/img/bm4.png)

- Result Matrix
- OkOperation: successful operations
Expand Down Expand Up @@ -199,7 +197,7 @@ We take the application of CRRC Qingdao Sifang Vehicle Research Institute Co., L

Test objective: Simulate the actual needs of switching time series databases in the scene of CRRC Qingdao Sifang Institute, and compare the performance of the expected IoTDB and KairosDB used by the original system.

测试环境:为了保证在实验过程中消除其他无关服务与进程对数据库性能的影响,以及不同数据库之间的相互影响,本实验中的本地数据库均部署并运行在资源配置相同的多个独立的虚拟机上。因此,本实验搭建了 4 台 Linux( CentOS7 /x86) 虚拟机,并分别在上面部署了IoT-benchmark、 IoTDB数据库、KairosDB数据库、MySQL数据库。每一台虚拟机的具体资源配置如表2-1所示。每一台虚拟机的具体用途如表2-2所示。Test environment: In order to ensure that the impact of other irrelevant services and processes on database performance and the mutual influence between different databases are eliminated during the experiment, the local databases in this experiment are deployed and run on multiple independent virtual servers with the same resource configuration. Therefore, this experiment set up 4 Linux (CentOS7 /x86) virtual machines, and deployed IoT-benchmark, IoTDB database, KairosDB database, and MySQL database on them respectively. The specific resource configuration of each virtual machine is shown in Table 2-1. The specific usage of each virtual machine is shown in Table 2-2.
Test environment: In order to ensure that the impact of other irrelevant services and processes on database performance and the mutual influence between different databases are eliminated during the experiment, the local databases in this experiment are deployed and run on multiple independent virtual servers with the same resource configuration. Therefore, this experiment set up 4 Linux (CentOS7 /x86) virtual machines, and deployed IoT-benchmark, IoTDB database, KairosDB database, and MySQL database on them respectively. The specific resource configuration of each virtual machine is shown in Table 2-1. The specific usage of each virtual machine is shown in Table 2-2.

Table 2-1 Virtual machine configuration information

Expand Down Expand Up @@ -253,39 +251,39 @@ Table 2-3 Configuration parameter information

First, start the tested time series databases Apache-IoTDB and KairosDB on 172.21.4.3 and 172.21.4.4 respectively, and then start server resource monitoring through the ser-benchamrk.sh script on 172.21.4.2, 172.21.4.3 and 172.21.4.4 (Figure 2-1). Then modify the conf/config.properties files in the iotdb-0.13-0.0.1 and kairosdb-0.0.1 folders in 172.21.4.2 according to Table 2-3 to meet the test requirements. Use benchmark.sh to start the writing test of Apache-IoTDB and KairosDB successively.

![img](https://alioss.timecho.com/docs/img/5.png)
![img](https://alioss.timecho.com/docs/img/bm5.png)

Figure 2-1 Server monitoring tasks

For example, if we first start the test on KairosDB, IoT-benchmark will create a CONFIG data table in the MySQL database to store the configuration information of this test (Figure 2-2), and there will be a log output of the current test progress during the test execution (Figure 2-3) . When the test is completed, the test result will be output (Figure 2-3), and the result will be written into the FINAL_RESULT data table (Figure 2-4).

![img](https://alioss.timecho.com/docs/img/6.png)
![img](https://alioss.timecho.com/docs/img/bm6.png)

Figure 2-2 Test configuration information table

![img](https://alioss.timecho.com/docs/img/7.png)
![img](https://alioss.timecho.com/docs/img/8.png)
![img](https://alioss.timecho.com/docs/img/9.png)
![img](https://alioss.timecho.com/docs/img/10.png)
![img](https://alioss.timecho.com/docs/img/bm7.png)
![img](https://alioss.timecho.com/docs/img/bm8.png)
![img](https://alioss.timecho.com/docs/img/bm9.png)
![img](https://alioss.timecho.com/docs/img/bm10.png)

Figure 2-3 Test progress and results

![img](https://alioss.timecho.com/docs/img/11.png)
![img](https://alioss.timecho.com/docs/img/bm11.png)

Figure 2-4 Test result table

Afterwards, we will start the test on Apache-IoTDB. The same IoT-benchmark will write the test configuration information in the MySQL database CONFIG data table. During the test execution, there will be a log to output the current test progress. When the test is completed, the test result will be output, and the result will be written into the FINAL_RESULT data table.

依照测试结果信息我们知道同样的配置写入Apache-IoTDB和KairosDB写入延时时间分别为:55.98ms和1324.45ms;写入吞吐分别为:5,125,600.86点/秒和224,819.01点/秒;测试分别执行了585.30秒和11777.99秒。并且KairosDB有写入失败出现,排查后发现是数据磁盘使用率已达到100%,无磁盘空间继续接收数据。而Apache-IoTDB无写入失败现象,全部数据写入完毕后占用磁盘空间仅为4.7G(如图2-5所示);从写入吞吐和磁盘占用情况上看Apache-IoTDB均优于KairosDB。当然后续还有其他测试来从多方面观察和对比,比如查询性能、文件压缩比、数据安全性等。According to the test result information, we know that under the same configuration the write delay times of Apache-IoTDB and KairosDB are 55.98ms and 1324.45ms respectively; the write throughputs are 5,125,600.86 points/second and 224,819.01 points/second respectively; the tests were executed respectively 585.30 seconds and 11777.99 seconds. And KairosDB has a write failure. After investigation, it is found that the data disk usage has reached 100%, and there is no disk space to continue receiving data. However, Apache-IoTDB has no write failure, and the disk space occupied after all data is written is only 4.7G (as shown in Figure 2-5); Apache-IoTDB is better than KairosDB in terms of write throughput and disk occupation. Of course, there will be other tests in the follow-up to observe and compare from various aspects, such as query performance, file compression ratio, data security, etc.
According to the test result information, we know that under the same configuration the write delay times of Apache-IoTDB and KairosDB are 55.98ms and 1324.45ms respectively; the write throughputs are 5,125,600.86 points/second and 224,819.01 points/second respectively; the tests were executed respectively 585.30 seconds and 11777.99 seconds. And KairosDB has a write failure. After investigation, it is found that the data disk usage has reached 100%, and there is no disk space to continue receiving data. However, Apache-IoTDB has no write failure, and the disk space occupied after all data is written is only 4.7G (as shown in Figure 2-5); Apache-IoTDB is better than KairosDB in terms of write throughput and disk occupation. Of course, there will be other tests in the follow-up to observe and compare from various aspects, such as query performance, file compression ratio, data security, etc.

![img](https://alioss.timecho.com/docs/img/12.png)
![img](https://alioss.timecho.com/docs/img/bm12.png)

Figure 2-5 Disk usage

So what is the resource usage of each server during the test? What is the specific performance of each write operation? At this time, we can visualize the data in the server monitoring table and test process recording table by installing and using Tableau. The use of Tableau will not be introduced in this article. After connecting to the data table for test data persistence, the specific results are as follows (taking Apache-IoTDB as an example):

![img](https://alioss.timecho.com/docs/img/13.png)
![img](https://alioss.timecho.com/docs/img/14.png)
![img](https://alioss.timecho.com/docs/img/bm13.png)
![img](https://alioss.timecho.com/docs/img/bm14.png)

Figure 2-6 Visualization of testing process in Tableau

Expand All @@ -310,7 +308,7 @@ Table 2-4 Configuration parameter information

Results:

![img](https://alioss.timecho.com/docs/img/15.png)
![img](https://alioss.timecho.com/docs/img/bm15.png)

Figure 2-7 Query test results

Expand Down
Loading

0 comments on commit 992aa87

Please sign in to comment.