Skip to content

Commit

Permalink
[doc](tvf)partition tvf and resource in s3 tvf (#869)
Browse files Browse the repository at this point in the history
- partitions tvf
- Permission instructions for using resources in S3 TVF

---------

Co-authored-by: Luzhijing <[email protected]>
Co-authored-by: morningman <[email protected]>
  • Loading branch information
3 people authored Aug 13, 2024
1 parent 157548f commit f77d02e
Show file tree
Hide file tree
Showing 135 changed files with 12,180 additions and 138 deletions.
18 changes: 9 additions & 9 deletions docs/data-operate/import/load-data-format.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,8 @@ The following import methods support data import in CSV format:
- [Broker Load](./import-way/broker-load-manual.md)
- [Routine Load](./import-way/routine-load-manual.md)
- [MySQL Load](./import-way/mysql-load-manual.md)
- [INSERT INTO FROM S3 TVF](../../sql-manual/sql-functions/table-functions/s3)
- [INSERT INTO FROM HDFS TVF](../../sql-manual/sql-functions/table-functions/hdfs)
- [INSERT INTO FROM S3 TVF](../../sql-manual/sql-functions/table-valued-functions/s3)
- [INSERT INTO FROM HDFS TVF](../../sql-manual/sql-functions/table-valued-functions/hdfs)

### Supported CSV Formats
- csv: File without header and type
Expand Down Expand Up @@ -136,8 +136,8 @@ Currently, only the following import methods support data import in JSON format:
- [Stream Load](./import-way/stream-load-manual.md)
- [Broker Load](./import-way/broker-load-manual.md)
- [Routine Load](./import-way/routine-load-manual.md)
- [INSERT INTO FROM S3 TVF](../../sql-manual/sql-functions/table-functions/s3)
- [INSERT INTO FROM HDFS TVF](../../sql-manual/sql-functions/table-functions/hdfs)
- [INSERT INTO FROM S3 TVF](../../sql-manual/sql-functions/table-valued-functions/s3)
- [INSERT INTO FROM HDFS TVF](../../sql-manual/sql-functions/table-valued-functions/hdfs)

### Supported JSON Formats

Expand Down Expand Up @@ -725,8 +725,8 @@ For Kafka data sources, the content in each Massage is treated as a complete JSO
The following import methods support importing data in CSV format:
- [Stream Load](./import-way/stream-load-manual.md)
- [Broker Load](./import-way/broker-load-manual.md)
- [INSERT INTO FROM S3 TVF](../../sql-manual/sql-functions/table-functions/s3)
- [INSERT INTO FROM HDFS TVF](../../sql-manual/sql-functions/table-functions/hdfs)
- [INSERT INTO FROM S3 TVF](../../sql-manual/sql-functions/table-valued-functions/s3)
- [INSERT INTO FROM HDFS TVF](../../sql-manual/sql-functions/table-valued-functions/hdfs)
### Import Examples
Expand Down Expand Up @@ -762,8 +762,8 @@ WITH S3
The following import methods support importing data in CSV format:
- [Stream Load](./import-way/stream-load-manual.md)
- [Broker Load](./import-way/broker-load-manual.md)
- [INSERT INTO FROM S3 TVF](../../sql-manual/sql-functions/table-functions/s3)
- [INSERT INTO FROM HDFS TVF](../../sql-manual/sql-functions/table-functions/hdfs)
- [INSERT INTO FROM S3 TVF](../../sql-manual/sql-functions/table-valued-functions/s3)
- [INSERT INTO FROM HDFS TVF](../../sql-manual/sql-functions/table-valued-functions/hdfs)
### Import Examples
Expand Down Expand Up @@ -792,4 +792,4 @@ WITH S3
"AWS_SECRET_KEY"="AWS_SECRET_KEY",
"AWS_REGION" = "AWS_REGION"
);
```
```
2 changes: 1 addition & 1 deletion docs/lakehouse/datalake-analytics/iceberg.md
Original file line number Diff line number Diff line change
Expand Up @@ -268,4 +268,4 @@ You can use the `FOR TIME AS OF` and `FOR VERSION AS OF` statements to read hist

`SELECT * FROM iceberg_tbl FOR VERSION AS OF 868895038966572;`

In addition, you can use the [iceberg_meta](../../sql-manual/sql-functions/table-functions/iceberg-meta.md) table function to query the snapshot information of the specified table.
In addition, you can use the [iceberg_meta](../../sql-manual/sql-functions/table-valued-functions/iceberg-meta.md) table function to query the snapshot information of the specified table.
6 changes: 4 additions & 2 deletions docs/lakehouse/file.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,11 @@ With the Table Value Function feature, Doris is able to query files in object st

For more usage details, please see the documentation:

* [S3](https://doris.apache.org/docs/dev/sql-manual/sql-functions/table-functions/s3/): supports file analysis on object storage compatible with S3
* [S3](../sql-manual/sql-functions/table-valued-functions/s3.md): supports file analysis on object storage compatible with S3

* [HDFS](https://doris.apache.org/docs/dev/sql-manual/sql-functions/table-functions/hdfs/): supports file analysis on HDFS
* [HDFS](../sql-manual/sql-functions/table-valued-functions/hdfs.md): supports file analysis on HDFS

* [LOCAL](../sql-manual/sql-functions/table-valued-functions/local.md): supports file analysis on local file system

The followings illustrate how file analysis is conducted with the example of S3 Table Value Function.

Expand Down
10 changes: 5 additions & 5 deletions docs/query/view-materialized-view/async-materialized-view.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ Specific syntax can be viewed [CREATE ASYNC MATERIALIZED VIEW](../../sql-manual/
select * from mv_infos("database"="tpch") where Name="mv1";
```

The unique features of materialized views can be viewed through [mv_infos()](../../sql-manual/sql-functions/table-functions/mv_infos.md)
The unique features of materialized views can be viewed through [mv_infos()](../../sql-manual/sql-functions/table-valued-functions/mv_infos.md)

Properties related to table, still viewed through [SHOW TABLES](../../sql-manual/sql-statements/Show-Statements/SHOW-TABLES.md)

Expand Down Expand Up @@ -142,7 +142,7 @@ Task is used to describe specific refresh information, such as the time used for
select * from jobs("type"="mv") order by CreateTime;
```

Specific syntax can be viewed [jobs("type"="mv")](../../sql-manual/sql-functions/table-functions/jobs.md)
Specific syntax can be viewed [jobs("type"="mv")](../../sql-manual/sql-functions/table-valued-functions/jobs.md)

#### Pause materialized view job scheduled scheduling

Expand Down Expand Up @@ -170,7 +170,7 @@ Specific syntax can be viewed [RESUME MATERIALIZED VIEW JOB](../../sql-manual/sq
select * from tasks("type"="mv");
```

Specific syntax can be viewed [tasks("type"="mv")](../../sql-manual/sql-functions/table-functions/tasks.md)
Specific syntax can be viewed [tasks("type"="mv")](../../sql-manual/sql-functions/table-valued-functions/tasks.md)

#### Cancel the task of objectifying the view

Expand Down Expand Up @@ -536,15 +536,15 @@ The commonly used commands for `olapTable` are also applicable to materialized v

The unique commands for materialized views mainly include the following:
#### View materialized view metadata
[mv_infos()](../../sql-manual/sql-functions/table-functions/mv_infos)
[mv_infos()](../../sql-manual/sql-functions/table-valued-functions/mv_infos)

Focus on the following fields:
- State: If the state changes to SCHEMA_CHANGE, it means the schema of the base table has changed. In this case, the materialized view cannot be used for transparent rewriting (but direct querying of the materialized view is not affected). If the next refresh task is successful, the state will be restored to NORMAL.
- SchemaChangeDetail: The reason for the SCHEMA_CHANGE.
- RefreshState: The status of the last refresh task of the materialized view. If it is FAIL, it means the execution failed, and further localization can be done through tasks().
- SyncWithBaseTables: Whether the materialized view is synchronized with the base table data. If not synchronized, further determination can be made by using show partitions to identify which partition is not synchronized.
#### View tasks for the materialized view
[tasks("type"="mv")](../../sql-manual/sql-functions/table-functions/tasks.md)
[tasks("type"="mv")](../../sql-manual/sql-functions/table-valued-functions/tasks.md)

Focus on the following fields:
- Status: If it is FAILED, it means the task execution failed. You can check the reason for failure through ErrorMsg. You can also search Doris logs using LastQueryId to get more detailed error information.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
{
"title": "frontends_disks",
"title": "FRONTENDS_DISKS",
"language": "en"
}
---
Expand Down Expand Up @@ -84,4 +84,4 @@ mysql> select * from frontends_disk()\G

### keywords

frontends_disks
frontends_disks
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,11 @@ File format parameters:
other kinds of parameters:

- `path_partition_keys`: (optional) Specifies the column names carried in the file path. For example, if the file path is /path/to/city=beijing/date="2023-07-09", you should fill in `path_partition_keys="city,date"`. It will automatically read the corresponding column names and values from the path during load process.
- `resource`:(optional)Specify the resource name. Hdfs Tvf can use the existing Hdfs resource to directly access Hdfs. You can refer to the method for creating an Hdfs resource: [CREATE-RESOURCE](../../sql-statements/Data-Definition-Statements/Create/CREATE-RESOURCE.md). This property is supported starting from version 2.1.4 .
- `resource`:(optional)Specify the resource name. Hdfs Tvf can use the existing Hdfs resource to directly access Hdfs. You can refer to the method for creating an Hdfs resource: [CREATE-RESOURCE](../../sql-statements/Data-Definition-Statements/Create/CREATE-RESOURCE.md). This property is supported starting from version 2.1.4.

:::tip Tip
To directly query a TVF or create a VIEW based on that TVF, you need to have usage permission for that resource. To query a VIEW created based on TVF, you only need select permission for that VIEW.
:::

### Examples

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,17 +27,17 @@ under the License.
## `jobs`

### Name
:::tip

jobs
- since 2.1
:::

### description

Table function, generating a temporary task table, which can view job information in a certain task type.

This function is used in the from clause.

This function is supported since 2.1.0.

#### syntax

`jobs("type"="")`
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
{
"title": "local",
"title": "LOCAL",
"language": "en"
}
---
Expand All @@ -24,7 +24,7 @@ specific language governing permissions and limitations
under the License.
-->

## Local
## local

### Name

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ Table function, generating temporary tables for asynchronous materialized views,

This function is used in the from clause.

This funciton is supported since 2.1.0.

#### syntax

`mv_infos("database"="")`
Expand Down
130 changes: 130 additions & 0 deletions docs/sql-manual/sql-functions/table-valued-functions/partitions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
---
{
"title": "PARTITIONS",
"language": "en"
}
---

<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

## `partitions`

### Name

partitions

### Description

The table function generates a temporary partition TABLE, which allows you to view the PARTITION list of a certain TABLE.

This function is used in the from clause.

This function is supported since 2.1.5

#### Syntax

`partitions("catalog"="","database"="","table"="")`

partitions() Table structure:
```sql
mysql> desc function partitions("catalog"="internal","database"="zd","table"="user");
+--------------------------+---------+------+-------+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+--------------------------+---------+------+-------+---------+-------+
| PartitionId | BIGINT | No | false | NULL | NONE |
| PartitionName | TEXT | No | false | NULL | NONE |
| VisibleVersion | BIGINT | No | false | NULL | NONE |
| VisibleVersionTime | TEXT | No | false | NULL | NONE |
| State | TEXT | No | false | NULL | NONE |
| PartitionKey | TEXT | No | false | NULL | NONE |
| Range | TEXT | No | false | NULL | NONE |
| DistributionKey | TEXT | No | false | NULL | NONE |
| Buckets | INT | No | false | NULL | NONE |
| ReplicationNum | INT | No | false | NULL | NONE |
| StorageMedium | TEXT | No | false | NULL | NONE |
| CooldownTime | TEXT | No | false | NULL | NONE |
| RemoteStoragePolicy | TEXT | No | false | NULL | NONE |
| LastConsistencyCheckTime | TEXT | No | false | NULL | NONE |
| DataSize | TEXT | No | false | NULL | NONE |
| IsInMemory | BOOLEAN | No | false | NULL | NONE |
| ReplicaAllocation | TEXT | No | false | NULL | NONE |
| IsMutable | BOOLEAN | No | false | NULL | NONE |
| SyncWithBaseTables | BOOLEAN | No | false | NULL | NONE |
| UnsyncTables | TEXT | No | false | NULL | NONE |
+--------------------------+---------+------+-------+---------+-------+
20 rows in set (0.02 sec)
```

* PartitionId:partition id
* PartitionName:partition name
* VisibleVersion:visible version
* VisibleVersionTime:visible version time
* State:state
* PartitionKey:partition key
* Range:range
* DistributionKey:distribution key
* Buckets:bucket num
* ReplicationNum:replication num
* StorageMedium:storage medium
* CooldownTime:cooldown time
* RemoteStoragePolicy:remote storage policy
* LastConsistencyCheckTime:last consistency check time
* DataSize:data size
* IsInMemory:is in memory
* ReplicaAllocation:replica allocation
* IsMutable:is mutable
* SyncWithBaseTables:Is it synchronized with the base table data (for partitioning asynchronous materialized views)
* UnsyncTables:Which base table data is not synchronized with (for partitions of asynchronous materialized views)

```sql
mysql> desc function partitions("catalog"="hive","database"="zdtest","table"="com2");
+-----------+------+------+-------+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------+------+-------+---------+-------+
| Partition | TEXT | No | false | NULL | NONE |
+-----------+------+------+-------+---------+-------+
1 row in set (0.11 sec)
```

* Partition:partition name

### Example

1. View the partition list of table1 under db1 in the internal catalog

```sql
mysql> select * from partitions("catalog"="internal","database"="db1","table"="table1");
```

2. View the partition information with partition name partition1 under table1

```sql
mysql> select * from partitions("catalog"="internal","database"="db1","table"="table1") where PartitionName = "partition1";
```

3. View the partition ID with the partition name 'partition1' under Table 1

```sql
mysql> select PartitionId from partitions("catalog"="internal","database"="db1","table"="table1") where PartitionName = "partition1";
```

### Keywords

partitions
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,11 @@ The following 2 parameters are used for loading in csv format
other parameter:

- `path_partition_keys`: (optional) Specifies the column names carried in the file path. For example, if the file path is /path/to/city=beijing/date="2023-07-09", you should fill in `path_partition_keys="city,date"`. It will automatically read the corresponding column names and values from the path during load process.
- `resource`:(optional)Specify the resource name. S3 tvf can use the existing S3 resource to directly access S3. You can refer to the method for creating an S3 resource: [CREATE-RESOURCE](../../sql-statements/Data-Definition-Statements/Create/CREATE-RESOURCE.md). This property is supported starting from version 2.1.4.
- `resource`:(optional)Specify the resource name. S3 tvf can use the existing S3 resource to directly access S3. You can refer to the method for creating an S3 resource: [CREATE-RESOURCE](../../sql-statements/Data-Definition-Statements/Create/CREATE-RESOURCE.md). This property is supported starting from version 2.1.4 .

:::tip Tip
To directly query a TVF or create a VIEW based on that TVF, you need to have usage permission for that resource. To query a VIEW created based on TVF, you only need select permission for that VIEW.
:::

### Example

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,8 @@ Table function, generates a temporary table of tasks, which allows you to view t

This function is used in the FROM clause.

This functions is supported since 2.1.0.

#### syntax

`tasks("type"="insert");`
Expand Down Expand Up @@ -169,4 +171,4 @@ mysql> select * from tasks("type"="mv") where JobName="inner_mtmv_75043";

### keywords

tasks, job, insert, mv, materilized view
tasks, job, insert, mv, materilized view
Original file line number Diff line number Diff line change
Expand Up @@ -65,8 +65,8 @@ Currently, only users with the ADMIN role can perform this operation.

#### Related Documentation

[PAUSE-JOB](../Alter/PAUSE-JOB.md),[RESUME-JOB](../Alter/RESUME-JOB.md),[DROP-JOB](../Drop/DROP-JOB.md), [QUERY-JOB](../../../sql-functions/table-functions/jobs.md),
[TVF-TASKS](../../../sql-functions/table-functions/tasks.md)
[PAUSE-JOB](../Alter/PAUSE-JOB.md),[RESUME-JOB](../Alter/RESUME-JOB.md),[DROP-JOB](../Drop/DROP-JOB.md), [QUERY-JOB](../../../sql-functions/table-valued-functions/jobs.md),
[TVF-TASKS](../../../sql-functions/table-valued-functions/tasks.md)

### Grammar

Expand Down Expand Up @@ -167,4 +167,4 @@ CREATE JOB my_job ON SCHEDULE EVERY 1 DAY STARTS '2020-01-01 00:00:00' ENDS '202

### Keywords

CREATE, JOB, SCHEDULE
CREATE, JOB, SCHEDULE
6 changes: 5 additions & 1 deletion i18n/zh-CN/docusaurus-plugin-content-docs/current.json
Original file line number Diff line number Diff line change
Expand Up @@ -295,6 +295,10 @@
"message": "表函数",
"description": "The label for category Table Functions in sidebar docs"
},
"sidebar.docs.category.Table Valued Functions": {
"message": "表值函数",
"description": "The label for category Table Valued Functions in sidebar docs"
},
"sidebar.docs.category.Analytic(Window) Functions": {
"message": "分析(窗口)函数",
"description": "The label for category Analytic(Window) Functions in sidebar docs"
Expand Down Expand Up @@ -427,4 +431,4 @@
"message": "使用教程",
"description": "The label for category BI and Database IDE in sidebar docs"
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,10 @@ hdfs(
其他参数:
- `path_partition_keys`:(选填)指定文件路径中携带的分区列名,例如/path/to/city=beijing/date="2023-07-09", 则填写`path_partition_keys="city,date"`,将会自动从路径中读取相应列名和列值进行导入。
- `resource`:(选填)指定 Resource 名,HDFS TVF 可以利用已有的 HFDS Resource 来直接访问 HDFS。创建 HDFS Resource 的方法可以参照 [CREATE-RESOURCE](../../sql-statements/Data-Definition-Statements/Create/CREATE-RESOURCE.md)。该功能自 2.1.4 版本开始支持。

:::tip 注意
直接查询 TVF 或基于该 TVF 创建 View ,需要拥有该 Resource 的 USAGE 权限,查询基于 TVF 创建的 View ,只需要该 View 的 SELECT 权限
:::

### Examples

Expand Down
Loading

0 comments on commit f77d02e

Please sign in to comment.