Skip to content

Commit

Permalink
integration databases
Browse files Browse the repository at this point in the history
  • Loading branch information
@wojcik-dorota
wojcik-dorota committed Feb 27, 2025
1 parent 20e0be0 commit 3732592
Show file tree
Hide file tree
Showing 4 changed files with 201 additions and 118 deletions.
31 changes: 18 additions & 13 deletions docs/products/clickhouse/howto/data-service-integration.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -133,10 +133,8 @@ Learn about [managed credentials integrations](/docs/products/clickhouse/concept
[Set up a managed-credentials integration](/docs/products/clickhouse/howto/data-service-integration#create-managed-credentials-integrations)
and
[create tables](/docs/products/clickhouse/howto/data-service-integration#create-tables)
where the data will
be available in Aiven for ClickHouse.
[Access tables](/docs/products/clickhouse/howto/data-service-integration#access-credentials-storage)
where your credentials are stored.
for the data to be made available through the integration.
[Access your stored credentials](/docs/products/clickhouse/howto/data-service-integration#access-credentials-storage).

### Create managed-credentials integrations

Expand All @@ -149,11 +147,11 @@ where your credentials are stored.
1. On the **Integrations** page, go to the **Data sources** section and click
**ClickHouse Credentials**.

The **ClickHouse credentials integration** wizard opens and shows a list of external data sources
or Aiven-managed data services available for integration, if any. If there are no data
sources to integrate with, the wizard allows you to create them either by clicking
**Create service** (for Aiven-managed sources) or **Add external endpoint** (for external
sources).
The **ClickHouse credentials integration** wizard opens and shows a list of external
data sources or Aiven-managed data services available for integration, if any. If there
are no data sources to integrate with, the wizard allows you to create them either by
clicking **Create service** (for Aiven-managed sources) or **Add external endpoint**
(for external sources).

1. In the **ClickHouse credentials integration** wizard:

Expand All @@ -168,6 +166,13 @@ where your credentials are stored.

1. Click **Enable integration**.
1. Optionally, click **Test connection** > **Open in query editor** > **Execute**.

:::note[Alternative]
You can test the connection any time later by going to your Aiven for ClickHouse
service's **Integrations** page, finding the credentials integration, and clicking
<ConsoleLabel name="actions"/> > <ConsoleLabel name="testconnection"/>.
:::

1. Click **Close**.

### Create tables
Expand Down Expand Up @@ -236,8 +241,8 @@ error message related to grants.
integrations for.
1. On the service's page, go to one of the following:

- <ConsoleLabel name="overview"/> > **Integrations**
- <ConsoleLabel name="integrations"/>
- <ConsoleLabel name="overview"/> in the sidebar > **Integrations**
- <ConsoleLabel name="integrations"/> in the sidebar

## Stop data source integrations

Expand All @@ -252,10 +257,10 @@ all databases and configuration information from Aiven for ClickHouse.
want to stop integrations for.
1. On the service's page, take one of the following courses of action:

- On the <ConsoleLabel name="overview"/> page, go to **Integrations**, find an
- Click <ConsoleLabel name="overview"/> > **Integrations**, find an
integration to be stopped, and click <ConsoleLabel name="actions"/> >
<ConsoleLabel name="disconnect"/>.
- Go to the <ConsoleLabel name="integrations"/> page, find an integration to be stopped,
- Click <ConsoleLabel name="integrations"/>, find an integration to be stopped,
and click <ConsoleLabel name="actions"/> > <ConsoleLabel name="disconnect"/>.

Your integration is terminated and all the corresponding databases and configuration
Expand Down
247 changes: 144 additions & 103 deletions docs/products/clickhouse/howto/integration-databases.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,149 +4,190 @@ title: Manage Aiven for ClickHouse® integration databases

import ConsoleLabel from "@site/src/components/non-swizzled/ConsoleIcons";
import RelatedPages from "@site/src/components/non-swizzled/RelatedPages";
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

You can set up and manage integration databases from the the <ConsoleLabel name="databasesandtables"/> view of your Aiven for ClickHouse service.
Set up and manage Aiven for ClickHouse® integration databases to access data hosted by
services you're integrated with.

By adding integrations databases in Aiven for ClickHouse, you create
streaming data pipelines across services. From Aiven for ClickHouse, you
can add integration databases connecting to Aiven for Kafka® and Aiven
for PostgreSQL®.
can add integration databases connecting to:

Aiven for ClickHouse supports
[regular integrations](/docs/products/clickhouse/howto/list-integrations) and
[data service integrations](/docs/products/clickhouse/howto/data-service-integration).

You can create Aiven for ClickHouse® integrations databases in the
[Aiven Console](https://console.aiven.io/) either when
[creating a new data service integration](/docs/products/clickhouse/howto/integration-databases#create-integ-db) or from the the <ConsoleLabel name="databasesandtables"/> view of your
service.

For information on how to set up integration databases when creating a
new data service integration, see
[Manage Aiven for ClickHouse® data service integrations](/docs/products/clickhouse/howto/data-service-integration).
- Aiven for Kafka®
- Aiven for PostgreSQL®
- Aiven for AlloyDB Omni

## Prerequisites

- Aiven account
- Access to [Aiven Console](https://console.aiven.io/)
- Access to the [Aiven Console](https://console.aiven.io/)
- Running Aiven for ClickHouse service
- [Service integration](/docs/products/clickhouse/howto/data-service-integration) between
the Aiven for ClickHouse service and a data source service (Aiven for Kafka, Aiven for
PostgreSQL, or Aiven for AlloyDB Omni)

## Create integration databases {#create-integ-db}
## Create integration databases

:::note
You can create both PostgrSQL and Apache Kafka integration databases for
Aiven for ClickHouse. This instruction uses *PostgreSQL* as an example.
:::
Depending on your data source, select **PostgreSQL**, **Apache Kafka**, or **AlloyDB Omni**.

1. Log in to the [Aiven Console](https://console.aiven.io/).
1. In the <ConsoleLabel name="Services"/> page, select an Aiven for ClickHouse service you
want to add integration databases to.
1. In your service's page, select <ConsoleLabel name="databasesandtables"/> from the
<Tabs groupId="group1">
<TabItem value="pg" label="PostgreSQL" default>
1. Log in to the [Aiven Console](https://console.aiven.io/), and go to your project.
1. On the <ConsoleLabel name="Services"/> page, select an Aiven for ClickHouse service.
1. On your service's page, click <ConsoleLabel name="databasesandtables"/> in the
sidebar.
1. In the <ConsoleLabel name="databasesandtables"/> view, select **Create database** >
1. On the **Databases and tables** page, click **Create database** >
**PostgreSQL integration database**.
1. In **Create PostgreSQL integration database** wizard, select one of
the following options:
- To add an integration database to a service that is not yet
integrated, go to the **New data service integration** tab.

<details><summary>
Expand for next steps
</summary>
1. In the **Create PostgreSQL integration database** wizard:

1. Select a service from the list of services available for
integration.
1. Select **Continue**.
1. In the **Add integration databases** section, enter database
names and schema names and select **Integrate & Create**
when ready.
1. Select a service to be a data source for the integration database, and click
**Continue**.
1. Enter **Database name** and **Schema name**, and click **Save**.

You can preview the created databases by selecting **Databases
and tables** from the sidebar.

</details>
</TabItem>
<TabItem value="kafka" label="Apache Kafka">
1. Log in to the [Aiven Console](https://console.aiven.io/), and go to your project.
1. On the <ConsoleLabel name="Services"/> page, select an Aiven for ClickHouse service.
1. On your service's page, click <ConsoleLabel name="databasesandtables"/> in the
sidebar.
1. On the **Databases and tables** page, click **Create database** >
**Kafka integration database**.
1. In the **Create Kafka integration database** wizard:

1. Select a service to be a data source for the integration database, and click
**Continue**.
1. Set up the integration table details:
- Enter a table name.
- Enter a consumer group name.
- Select topics.
- Select a data format.
- Enter table columns.
1. Click **Save table details** > **Save**.

</TabItem>
<TabItem value="alloydb" label="AlloyDB Omni">
1. Log in to the [Aiven Console](https://console.aiven.io/), and go to your project.
1. On the <ConsoleLabel name="Services"/> page, select an Aiven for ClickHouse service.
1. On your service's page, click <ConsoleLabel name="databasesandtables"/> in the
sidebar.
1. On the **Databases and tables** page, click **Create database** >
**AlloyDB Omni integration database**.
1. In the **Create AlloyDB Omni integration database** wizard:

- To add an integration database to an already integrated service,
go to the **Existing integration** tab.
1. Select a service to be a data source for the integration database, and click
**Continue**.
1. Enter **Database name** and **Schema name**, and click **Save**.

<details><summary>
Expand for next steps
</summary>
</TabItem>
</Tabs>

1. Select a service from the list of integrated services.
1. Select **Continue**.
1. In the **Add integration databases** section, enter database
names and schema names and select **Create** when ready.
## View integration databases or tables

You can preview the created databases by selecting **Databases
and tables** from the sidebar.
To preview integration databases or tables for your Aiven for ClickHouse service:

</details>
1. Log in to the [Aiven Console](https://console.aiven.io/), and go to your project.
1. On the <ConsoleLabel name="Services"/> page, select an Aiven for ClickHouse service to
preview data structures for.
1. On your service's page, click <ConsoleLabel name="databasesandtables"/> in the
sidebar.
1. Find your integration database, and expand it using <ConsoleLabel name="downarrow"/>
to display tables inside it.
1. Find your table, and click <ConsoleLabel name="actions"/> > <ConsoleLabel name="viewdetails"/>.

## View integration databases
## Edit integration databases or tables

1. Log in to the [Aiven Console](https://console.aiven.io/).
1. In the <ConsoleLabel name="Services"/> page, select an Aiven for ClickHouse service you
want to check integration databases for.
1. In your service's page, select <ConsoleLabel name="databasesandtables"/> from the
sidebar to discover your integration databases in the **Databases
and tables** list.
Add or delete integration databases or tables. Update table details.

:::note
PostgreSQL is currently only supported as a source.
You cannot edit tables inside PostgreSQL integration databases.
:::

## Edit integration databases
Depending on what you intend to edit, select **Database** or **Table**.

:::note
You can only edit Apache Kafka integration databases and tables.
:::

1. Log in to the [Aiven Console](https://console.aiven.io/).
<Tabs groupId="group1">
<TabItem value="db" label="Database" default>

1. In the <ConsoleLabel name="Services"/> page, select an Aiven for ClickHouse service you
want to edit integration databases for.
1. Log in to the [Aiven Console](https://console.aiven.io/), and go to your project.
1. On the <ConsoleLabel name="Services"/> page, select an Aiven for ClickHouse service
that includes a database to be edited.
1. On your service's page, click <ConsoleLabel name="databasesandtables"/> in the
sidebar.
1. Find the integration database to be edited, and click <ConsoleLabel name="actions"/> >
<ConsoleLabel name="editdatabase"/>.
1. Add or delete integration databases using <ConsoleLabel name="add"/> or
<ConsoleLabel name="delete"/> respectively, and click **Save**.

:::note[Alternative for PostgreSQL]
Go to your Aiven for ClickHouse service's **Integrations** page,
find an integration, and clicking click <ConsoleLabel name="actions"/> >
<ConsoleLabel name="editdatabase"/>.
:::

1. In your service's page, select <ConsoleLabel name="databasesandtables"/> from the
sidebar to find the **Databases and tables** list.
</TabItem>
<TabItem value="tb" label="Table">

1. Find an Apache Kafka integration database to edit on the **Databases and tables** list,
and click <ConsoleLabel name="actions"/> > <ConsoleLabel name="editdatabase"/>.
#### Add or delete tables

1. In the **Edit database** wizard, find a table to edit
in the **Configured tables** list and expand its details by
selecting the angle brackets icon.
1. Log in to the [Aiven Console](https://console.aiven.io/), and go to your project.
1. On the <ConsoleLabel name="Services"/> page, select an Aiven for ClickHouse service
that includes a database where you intend to add or delete tables.
1. On your service's page, click <ConsoleLabel name="databasesandtables"/> in the
sidebar.
1. Find the database, and click <ConsoleLabel name="actions"/> >
<ConsoleLabel name="editdatabase"/>.
1. Add or delete tables using <ConsoleLabel name="addtable"/> or
<ConsoleLabel name="delete"/> respectively, and click **Save**.

:::note
You can also create a table for the database you are editing by
selecting **Add another table**.
:::
#### Update table details

1. In the table details section, update any of the following fields:
1. Log in to the [Aiven Console](https://console.aiven.io/), and go to your project.
1. On the <ConsoleLabel name="Services"/> page, select an Aiven for ClickHouse service
that includes a table to be edited.
1. On your service's page, click <ConsoleLabel name="databasesandtables"/> in the
sidebar.
1. Find a database including the table to be edited, and expand the database using
<ConsoleLabel name="downarrow"/> to display tables inside it.
1. Find the table to be edited, and click <ConsoleLabel name="actions"/> >
<ConsoleLabel name="edittable"/>.
1. In the **Edit table** window, update table details, and click **Saves**.

- Table name
- Consumer group name
- Topics
- Data format
- Table columns
</TabItem>
</Tabs>

1. Select **Update table details** > **Save changes**.
## Delete integration databases or tables

Your integration database and/or its tables have been updated.
Depending on what you intend to delete, select **Database** or **Table**.

## Delete integration databases
<Tabs groupId="group1">
<TabItem value="db" label="Database" default>

1. Log in to the [Aiven Console](https://console.aiven.io/).
1. In the <ConsoleLabel name="Services"/> page, select an Aiven for ClickHouse service you
want to delete integration databases for.
1. In your service's page, select <ConsoleLabel name="databasesandtables"/> from the
sidebar to find the **Databases and tables** list.
1. In the **Databases and tables** list, find your integration database and click
1. Log in to the [Aiven Console](https://console.aiven.io/), and go to your project.
1. On the <ConsoleLabel name="Services"/> page, select an Aiven for ClickHouse service
where a database to be deleted resides.
1. On your service's page, click <ConsoleLabel name="databasesandtables"/> in the
sidebar.
1. On the **Databases and tables** page, find your integration database and click
<ConsoleLabel name="actions"/> > <ConsoleLabel name="deletedatabase"/>.
1. In the **Delete database confirmation** popup, study the impact and
select **Confirm** if you accept removing the database along with
the tables inside it.
1. In the **Delete database confirmation** window, study the impact and
click **Confirm** if you accept removing the database along with the tables inside it.

Your integration database has been removed from the **Databases and tables** list.
</TabItem>
<TabItem value="tb" label="Table">

1. Log in to the [Aiven Console](https://console.aiven.io/), and go to your project.
1. On the <ConsoleLabel name="Services"/> page, select an Aiven for ClickHouse service
where a table to be deleted resides.
1. On your service's page, click <ConsoleLabel name="databasesandtables"/> in the
sidebar.
1. Find the integration database including the table, and expand it using
<ConsoleLabel name="downarrow"/> to display tables inside it.
1. Find the table to be deleted, and click <ConsoleLabel name="actions"/> >
<ConsoleLabel name="deletetable"/>.
1. In the **Delete table confirmation** window, study the impact and click **Confirm** if
you accept removing the table.

</TabItem>
</Tabs>

<RelatedPages/>

Expand Down
4 changes: 2 additions & 2 deletions docs/products/clickhouse/howto/manage-databases-tables.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ using an SQL client such as the

:::tip
You can also create
[integration databases](/docs/products/clickhouse/howto/integration-databases#create-integ-db)
[integration databases](/docs/products/clickhouse/howto/integration-databases#create-integration-databases)
in Aiven for ClickHouse.
:::

Expand Down Expand Up @@ -170,4 +170,4 @@ To remove your table in the [Aiven Console](https://console.aiven.io/):

<RelatedPages/>

[Create Aiven for ClickHouse integration databases](/docs/products/clickhouse/howto/integration-databases#create-integ-db)
[Create Aiven for ClickHouse integration databases](/docs/products/clickhouse/howto/integration-databases#create-integration-databases)
Loading

0 comments on commit 3732592

Please sign in to comment.