Skip to content

Commit

Permalink
getting-started pulse (#89)
Browse files Browse the repository at this point in the history
Co-authored-by: aaryaattrey <[email protected]>
  • Loading branch information
aaryaattrey and aaryaattrey authored Mar 26, 2024
1 parent bb50918 commit 13608f6
Show file tree
Hide file tree
Showing 5 changed files with 44 additions and 15 deletions.
59 changes: 44 additions & 15 deletions docs/getting-started-pulse.md
Original file line number Diff line number Diff line change
@@ -1,37 +1,40 @@
# Getting Started
For getting started with Dozer Analytics, create a new account at https://cloud.getdozer.io.

1) Start by creating a new project.
## Setup Project
Once logged in, users are directed to the Pulse dashboard, offering the ability to create multiple projects tailored to their specific needs. Start by creating a new project.

![Creating a new Project](./getting_started_pulse/create_new.png).

2) Configure your project.
You ll be prompted to configure the project. You can do the following things using Dozer Analytics immediately.
## Configure Project

![Setup a new project](./getting_started_pulse/setup.png).
You will be prompted to configure the project. You can do the following things using Dozer Pulse immediately.

1) Start ingesting data by leveraging one of the two methods.
![Setup a new project](./getting_started_pulse/setup.png)


## Ingest Data


Start ingesting data by leveraging one of the two methods.

There are two ways developers can ingest data into Dozer.
- Event Based Approach
- Dozer Connectors via CDC
- [Event Based Approach](#event-based)
- [Dozer Connectors via CDC](#data-stores-data-lakes--files)

### Event Based
Developers can conveniently push data to Dozer using REST APIs which will be the easiest way to get started. Or if you are already leveraging Kafka `Dozer` can pull data from Kafka using Schema Registry for message parsging.
#### Event Based
Developers can conveniently push data to Dozer using REST APIs which will be the easiest way to get started. Or if you are already leveraging Kafka `Dozer` can pull data from Kafka using Schema Registry for message parsing.

Typically you would ingest transactional or high volume data using this method. Dozer and Clickhouse boast amazing ingestion speeds when compared to other tools.

### Data Stores, Data Lakes & Files

Dozer utilizes `Dozer Connectors` for keeping upto date with all the databases. It pulls data from various sources like databases, data lakes, and warehouses using Change Data Capture (CDC) and periodic polling mechanisms. This ensures up-to-date data ingestion in real-time or near-real-time.


### Ingest a sample Event
#### Ingest a sample Event
Click on the (+) Add Data Source and select Events. You can try ingesting a sample record using the ingest API.

![Setup a new project](./getting_started_pulse/ingest.png)

### Sample Event
#### Sample Event
```bash

curl -X POST \
Expand All @@ -47,10 +50,36 @@ Click on the (+) Add Data Source and select Events. You can try ingesting a samp
}'
```

#### Data Stores, Data Lakes & Files

Dozer utilizes `Dozer Connectors` for keeping upto date with all the databases. It pulls data from various sources like databases, data lakes, and warehouses using Change Data Capture (CDC) and periodic polling mechanisms. This ensures up-to-date data ingestion in real-time or near-real-time.

#### Ingest sample data from a Datastore

Click on the (+) Add Data Source and select a database (currently supporting Postgres and MySQL). Input the relevant connection details like host address, port, username, and password.

## Çreate an Endpoint
![Ingest Data](./getting_started_pulse/connect-pg.png)


Once the connection to the source database is confirmed, a list of all available tables within the connected database is presented. From this list, users have the option to manually select and adjust which tables are to be replicated to the ClickHouse sink for subsequent analytics processing.

![Select Tables](./getting_started_pulse/select-tables.png)

Once all the steps are completed, users can access all tables from multiple sources in the "Sources" section. Additionally, all created endpoints and views are conveniently displayed next to the sidebar.

Users can easily view data records and schema information for each source table for quick reference. For instance, the schema for the job_assignments table can be viewed as follows:

![View Data](./getting_started_pulse/view-data.png)

## Create an Endpoint

Now that you have access to a datasource, you can create an endpoint with a query.
- Click on the (+) Add Endpoint and input the endpoint name, description, and the SQL query you want to execute on the underlying Clickhouse database.

![Create Endpoint](./getting_started_pulse/create-endpt.png)

- Alterntatively, you can create an endpoint by executing the following curl command.

```bash

curl -X POST \
Expand Down
Binary file added docs/getting_started_pulse/connect-pg.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/getting_started_pulse/create-endpt.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/getting_started_pulse/select-tables.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/getting_started_pulse/view-data.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 13608f6

Please sign in to comment.