Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enh: TDengine modify taosBenchmark new query rule cases and add doc #29638

Merged
merged 11 commits into from
Jan 24, 2025
28 changes: 14 additions & 14 deletions docs/en/14-reference/02-tools/10-taosbenchmark.md
Original file line number Diff line number Diff line change
Expand Up @@ -246,13 +246,14 @@ The query performance test mainly outputs the QPS indicator of query request spe

``` bash
complete query with 3 threads and 10000 query delay avg: 0.002686s min: 0.001182s max: 0.012189s p90: 0.002977s p95: 0.003493s p99: 0.004645s SQL command: select ...
INFO: Total specified queries: 30000
INFO: Spend 26.9530 second completed total queries: 30000, the QPS of all threads: 1113.049
```

- The first line represents the percentile distribution of query execution and query request delay for each of the three threads executing 10000 queries. The SQL command is the test query statement
- The second line indicates that a total of 10000 * 3 = 30000 queries have been completed
- The third line indicates that the total query time is 26.9653 seconds, and the query rate per second (QPS) is 1113.049 times/second
- The second line indicates that the total query time is 26.9653 seconds, the total queries is 10000 * 3 = 30000, and the query rate per second (QPS) is 1113.049 times/second
- If the `continue_if_fail` option is set to `yes` in the query, the last line will output the number of failed requests and error rate, the format like "error + number of failed requests (error rate)"
- QPS = number of successful requests / time spent (in seconds)
- Error rate = number of failed requests / (number of successful requests + number of failed requests)

#### Subscription metrics

Expand Down Expand Up @@ -334,9 +335,9 @@ Parameters related to supertable creation are configured in the `super_tables` s

- **child_table_exists**: Whether the child table already exists, default is "no", options are "yes" or "no".

- **child_table_count**: Number of child tables, default is 10.
- **childtable_count**: Number of child tables, default is 10.

- **child_table_prefix**: Prefix for child table names, mandatory, no default value.
- **childtable_prefix**: Prefix for child table names, mandatory, no default value.

- **escape_character**: Whether the supertable and child table names contain escape characters, default is "no", options are "yes" or "no".

Expand Down Expand Up @@ -403,7 +404,7 @@ Specify the configuration parameters for tag and data columns in `super_tables`

- **min**: The minimum value for the data type of the column/tag. Generated values will be greater than or equal to the minimum value.

- **max**: The maximum value for the data type of the column/tag. Generated values will be less than the minimum value.
- **max**: The maximum value for the data type of the column/tag. Generated values will be less than the maximum value.

- **scalingFactor**: Floating-point precision enhancement factor, only effective when the data type is float/double, valid values range from 1 to 1000000 positive integers. Used to enhance the precision of generated floating points, especially when min or max values are small. This attribute enhances the precision after the decimal point by powers of 10: a scalingFactor of 10 means enhancing the precision by 1 decimal place, 100 means 2 places, and so on.

Expand Down Expand Up @@ -431,11 +432,9 @@ Specify the configuration parameters for tag and data columns in `super_tables`

- **create_table_thread_count** : The number of threads for creating tables, default is 8.

- **connection_pool_size** : The number of pre-established connections with the TDengine server. If not configured, it defaults to the specified number of threads.

- **result_file** : The path to the result output file, default is ./output.txt.

- **confirm_parameter_prompt** : A toggle parameter that requires user confirmation after a prompt to continue. The default value is false.
- **confirm_parameter_prompt** : A toggle parameter that requires user confirmation after a prompt to continue. The value can be "yes" or "no", by default "no".

- **interlace_rows** : Enables interleaved insertion mode and specifies the number of rows to insert into each subtable at a time. Interleaved insertion mode refers to inserting the specified number of rows into each subtable in sequence and repeating this process until all subtable data has been inserted. The default value is 0, meaning data is inserted into one subtable completely before moving to the next.
This parameter can also be configured in `super_tables`; if configured, the settings in `super_tables` take higher priority and override the global settings.
Expand Down Expand Up @@ -464,12 +463,12 @@ For other common parameters, see Common Configuration Parameters.

Configuration parameters for querying specified tables (can specify supertables, subtables, or regular tables) are set in `specified_table_query`.

- **mixed_query** "yes": `Mixed Query` "no": `Normal Query`, default is "no"
`Mixed Query`: All SQL statements in `sqls` are grouped by the number of threads, with each thread executing one group. Each SQL statement in a thread needs to perform `query_times` queries.
`Normal Query `: Each SQL in `sqls` starts `threads` and exits after executing `query_times` times. The next SQL can only be executed after all previous SQL threads have finished executing and exited.
Regardless of whether it is a `Normal Query` or `Mixed Query`, the total number of query executions is the same. The total number of queries = `sqls` * `threads` * `query_times`. The difference is that `Normal Query` starts `threads` for each SQL query, while ` Mixed Query` only starts `threads` once to complete all SQL queries. The number of thread startups for the two is different.
- `General Query`: Each SQL in `sqls` starts `threads` threads to query this SQL, Each thread exits after executing the `query_times` queries, and only after all threads executing this SQL have completed can the next SQL be executed.
The total number of queries(`General Query`) = the number of `sqls` * `query_times` * `threads`
- `Mixed Query` : All SQL statements in `sqls` are divided into `threads` groups, with each thread executing one group. Each SQL statement needs to execute `query_times` queries.
The total number of queries(`Mixed Query`) = the number of `sqls` * `query_times`

- **query_interval** : Query interval, in seconds, default is 0.
- **query_interval** : Query interval, in millisecond, default is 0.

- **threads** : Number of threads executing the SQL query, default is 1.

Expand All @@ -491,6 +490,7 @@ The thread mode of the super table query is the same as the `Normal Query` mode
- **sqls** :
- **sql** : The SQL command to execute, required; for supertable queries, keep "xxxx" in the SQL command, the program will automatically replace it with all subtable names of the supertable.
- **result** : File to save the query results, if not specified, results are not saved.
- **Note**: The maximum number of SQL arrays configured under SQL is 100.

### Configuration Parameters for Subscription Scenarios

Expand Down
43 changes: 24 additions & 19 deletions docs/zh/14-reference/02-tools/10-taosbenchmark.md
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,9 @@ INFO: Spend 26.9530 second completed total queries: 30000, the QPS of all thread
- 第一行表示 3 个线程每个线程执行 10000 次查询及查询请求延时百分位分布情况,`SQL command` 为测试的查询语句
- 第二行表示总共完成了 10000 * 3 = 30000 次查询总数
- 第三行表示查询总耗时为 26.9653 秒,每秒查询率(QPS)为:1113.049 次/秒
- 如果在查询中设置了 `continue_if_fail` 选项为 `yes`,在最后一行中会输出失败请求个数及错误率,格式 error + 失败请求个数 (错误率)
- QPS = 成功请求数量 / 花费时间(单位秒)
- 错误率 = 失败请求数量 /(成功请求数量 + 失败请求数量)

#### 订阅指标

Expand Down Expand Up @@ -207,12 +210,12 @@ INFO: Consumed total msgs: 3000, total rows: 30000000

插入场景下 `filetype` 必须设置为 `insert`,该参数及其它通用参数详见[通用配置参数](#通用配置参数)

- ** keep_trying ** : 失败后进行重试的次数,默认不重试。需使用 v3.0.9 以上版本。
- **keep_trying** : 失败后进行重试的次数,默认不重试。需使用 v3.0.9 以上版本。

- ** trying_interval ** : 失败重试间隔时间,单位为毫秒,仅在 keep_trying 指定重试后有效。需使用 v3.0.9 以上版本。
- ** childtable_from 和 childtable_to ** : 指定写入子表范围,开闭区间为 [childtable_from, childtable_to).
- **trying_interval** : 失败重试间隔时间,单位为毫秒,仅在 keep_trying 指定重试后有效。需使用 v3.0.9 以上版本。
- **childtable_from 和 childtable_to** : 指定写入子表范围,开闭区间为 [childtable_from, childtable_to).

- ** continue_if_fail ** : 允许用户定义失败后行为
- **continue_if_fail** : 允许用户定义失败后行为

“continue_if_fail”:  “no”, 失败 taosBenchmark 自动退出,默认行为
“continue_if_fail”: “yes”, 失败 taosBenchmark 警告用户,并继续写入
Expand All @@ -224,7 +227,7 @@ INFO: Consumed total msgs: 3000, total rows: 30000000

- **name** : 数据库名。

- **drop** : 数据库已存在时是否删除重建,可选项为 "yes" 或 "no", 默认为 “yes”
- **drop** : 数据库已存在时是否删除,可选项为 "yes" 或 "no", 默认为 “yes”

#### 流式计算相关配置参数

Expand All @@ -250,9 +253,9 @@ INFO: Consumed total msgs: 3000, total rows: 30000000

- **child_table_exists** : 子表是否已经存在,默认值为 "no",可选值为 "yes" 或 "no"。

- **child_table_count** : 子表的数量,默认值为 10。
- **childtable_count** : 子表的数量,默认值为 10。

- **child_table_prefix** : 子表名称的前缀,必选配置项,没有默认值。
- **childtable_prefix** : 子表名称的前缀,必选配置项,没有默认值。

- **escape_character** : 超级表和子表名称中是否包含转义字符,默认值为 "no",可选值为 "yes" 或 "no"。

Expand Down Expand Up @@ -319,7 +322,7 @@ INFO: Consumed total msgs: 3000, total rows: 30000000

- **min** : 数据类型的 列/标签 的最小值。生成的值将大于或等于最小值。

- **max** : 数据类型的 列/标签 的最大值。生成的值将小于最小值
- **max** : 数据类型的 列/标签 的最大值。生成的值将小于最大值

- **scalingFactor** : 浮点数精度增强因子,仅当数据类型是 float/double 时生效,有效值范围为 1 至 1000000 的正整数。用于增强生成浮点数的精度,特别是在 min 或 max 值较小的情况下。此属性按 10 的幂次增强小数点后的精度:scalingFactor 为 10 表示增强 1 位小数精度,100 表示增强 2 位,依此类推。

Expand All @@ -343,15 +346,13 @@ INFO: Consumed total msgs: 3000, total rows: 30000000

- **thread_count** : 插入数据的线程数量,默认为 8。

- **thread_bind_vgroup** : 写入时 vgroup 是否和写入线程绑定,绑定后可提升写入速度, 取值为 "yes" 或 "no",默认值为 “no”, 设置为 “no” 后与原来行为一致。 当设为 “yes” 时,如果 thread_count 数量大小写入数据库的 vgroups 数量, thread_count 自动调整为 vgroups 数量;如果 thread_count 数量小于 vgroups 数量,写入线程数量不做调整,一个线程写完一个 vgroup 数据后再写下一个,同时保持一个 vgroup 同时只能由一个线程写入的规则。
- **thread_bind_vgroup** : 写入时 vgroup 是否和写入线程绑定,绑定后可提升写入速度, 取值为 "yes" 或 "no",默认值为 “no”, 设置为 “no” 后与原来行为一致。 当设为 “yes” 时,如果 thread_count 大于写入数据库 vgroups 数量, thread_count 自动调整为 vgroups 数量;如果 thread_count 小于 vgroups 数量,写入线程数量不做调整,一个线程写完一个 vgroup 数据后再写下一个,同时保持一个 vgroup 同时只能由一个线程写入的规则。

- **create_table_thread_count** : 建表的线程数量,默认为 8。

- **connection_pool_size** : 预先建立的与 TDengine 服务端之间的连接的数量。若不配置,则与所指定的线程数相同。

- **result_file** : 结果输出文件的路径,默认值为 ./output.txt。

- **confirm_parameter_prompt** : 开关参数,要求用户在提示后确认才能继续。默认值为 false
- **confirm_parameter_prompt** : 开关参数,要求用户在提示后确认才能继续, 可取值 "yes" or "no"。默认值为 "no"

- **interlace_rows** : 启用交错插入模式并同时指定向每个子表每次插入的数据行数。交错插入模式是指依次向每张子表插入由本参数所指定的行数并重复这个过程,直到所有子表的数据都插入完成。默认值为 0, 即向一张子表完成数据插入后才会向下一张子表进行数据插入。
在 `super_tables` 中也可以配置该参数,若配置则以 `super_tables` 中的配置为高优先级,覆盖全局设置。
Expand Down Expand Up @@ -381,12 +382,16 @@ interval 控制休眠时间,避免持续查询慢查询消耗 CPU ,单位为

查询指定表(可以指定超级表、子表或普通表)的配置参数在 `specified_table_query` 中设置。

- **mixed_query** : 查询模式,取值 “yes” 为`混合查询`, "no" 为`正常查询` , 默认值为 “no”
`混合查询`:`sqls` 中所有 sql 按 `threads` 线程数分组,每个线程执行一组, 线程中每个 sql 都需执行 `query_times` 次查询
`正常查询`:`sqls` 中每个 sql 启动 `threads` 个线程,每个线程执行完 `query_times` 次后退出,下个 sql 需等待上个 sql 线程全部执行完退出后方可执行
不管 `正常查询` 还是 `混合查询` ,执行查询总次数是相同的 ,查询总次数 = `sqls` 个数 * `threads` * `query_times`, 区别是 `正常查询` 每个 sql 都会启动 `threads` 个线程,而 `混合查询` 只启动一次 `threads` 个线程执行完所有 SQL, 两者启动线程次数不一样。
- **mixed_query** : 查询模式
“yes” :`混合查询`
"no"(默认值) :`普通查询`
`普通查询`:`sqls` 中每个 sql 启动 `threads` 个线程查询此 sql, 执行完 `query_times` 次查询后退出,执行此 sql 的所有线程都完成后进入下一个 sql
`查询总次数` = `sqls` 个数 * `query_times` * `threads`

`混合查询`:`sqls` 中所有 sql 分成 `threads` 个组,每个线程执行一组, 每个 sql 都需执行 `query_times` 次查询
`查询总次数` = `sqls` 个数 * `query_times`

- **query_interval** : 查询时间间隔,单位是秒,默认值为 0。
- **query_interval** : 查询时间间隔,单位: millisecond,默认值为 0。

- **threads** : 执行查询 SQL 的线程数,默认值为 1。

Expand All @@ -406,9 +411,9 @@ interval 控制休眠时间,避免持续查询慢查询消耗 CPU ,单位为
- **threads** : 执行查询 SQL 的线程数,默认值为 1。

- **sqls** :
- **sql** : 执行的 SQL 命令,必填;对于超级表的查询 SQL,在 SQL 命令中保留 "xxxx",程序会自动将其替换为超级表的所有子表名。
替换为超级表中所有的子表名。
- **sql** : 执行的 SQL 命令,必填;对于超级表的查询 SQL,在 SQL 命令中必须保留 "xxxx",会替换为超级下所有子表名后再执行。
- **result** : 保存查询结果的文件,未指定则不保存。
- **限制项** : sqls 下配置 sql 数组最大为 100 个

### 订阅场景配置参数

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,16 +20,5 @@
"sql": "select last_row(*) from stb00_9 ",
"result": "./query_res1.txt"
}]
},
"super_table_query": {
"stblname": "stb1",
"query_interval":20,
"threads": 4,
"sqls": [
{
"sql": "select last_row(ts) from xxxx",
"result": "./query_res2.txt"
}
]
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
{
"filetype": "query",
"cfgdir": "/etc/taos",
"host": "127.0.0.1",
"port": 6030,
"user": "root",
"password": "taosdata",
"confirm_parameter_prompt": "no",
"databases": "db",
"query_times": 1,
"super_table_query": {
"stblname": "stb1",
"query_interval":20,
"threads": 4,
"sqls": [
{
"sql": "select last_row(ts) from xxxx",
"result": "./query_res2.txt"
}
]
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -22,17 +22,6 @@
"result": "./query_res1.txt"
}
]
},
"super_table_query": {
"stblname": "stb1",
"query_interval": 1,
"threads": 3,
"sqls": [
{
"sql": "select last_row(ts) from xxxx",
"result": "./query_res2.txt"
}
]
}
}

Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
{
"filetype": "query",
"cfgdir": "/etc/taos",
"host": "127.0.0.1",
"port": 6030,
"user": "root",
"password": "taosdata",
"confirm_parameter_prompt": "no",
"databases": "db",
"query_times": 2,
"query_mode": "rest",
"super_table_query": {
"stblname": "stb1",
"query_interval": 1,
"threads": 3,
"sqls": [
{
"sql": "select last_row(ts) from xxxx",
"result": "./query_res2.txt"
}
]
}
}

Original file line number Diff line number Diff line change
Expand Up @@ -22,16 +22,5 @@
"result": "./query_res1.txt"
}
]
},
"super_table_query": {
"stblname": "stb1",
"query_interval": 1,
"threads": 3,
"sqls": [
{
"sql": "select last_row(ts) from xxxx",
"result": "./query_res2.txt"
}
]
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
{
"filetype": "query",
"cfgdir": "/etc/taos",
"host": "127.0.0.1",
"port": 6030,
"user": "root",
"password": "taosdata",
"confirm_parameter_prompt": "no",
"databases": "db",
"query_times": 2,
"query_mode": "taosc",
"super_table_query": {
"stblname": "stb1",
"query_interval": 1,
"threads": 3,
"sqls": [
{
"sql": "select last_row(ts) from xxxx",
"result": "./query_res2.txt"
}
]
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,5 @@
"sql": "select count(*) from db.stb",
"result": "rest_query_specified"
}]
},
"super_table_query": {
"stblname": "stb",
"sqls": [
{
"sql": "select count(*) from xxxx",
"result": "rest_query_super"
}
]
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"filetype":"query",
"cfgdir": "/etc/taos",
"confirm_parameter_prompt": "no",
"databases": "db",
"query_mode": "rest",
"connection_pool_size": 10,
"response_buffer": 10000,
"super_table_query": {
"stblname": "stb",
"sqls": [
{
"sql": "select count(*) from xxxx",
"result": "rest_query_super"
}
]
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -18,16 +18,5 @@
"sql": "select count(*) from db.stb",
"result": "taosc_query_specified"
}]
},
"super_table_query": {
"stblname": "stb",
"query_interval": 1,
"concurrent": 1,
"sqls": [
{
"sql": "select count(*) from xxxx",
"result": "taosc_query_super"
}
]
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
{
"filetype":"query",
"cfgdir": "/etc/taos",
"host": "localhost",
"port": 6030,
"user": "root",
"password": "taosdata",
"confirm_parameter_prompt": "no",
"databases": "db",
"query_times": 1,
"reset_query_cache": "yes",
"super_table_query": {
"stblname": "stb",
"query_interval": 1,
"concurrent": 1,
"sqls": [
{
"sql": "select count(*) from xxxx",
"result": "taosc_query_super"
}
]
}
}
Loading