Skip to content

Commit

Permalink
Added Redshift docs
Browse files Browse the repository at this point in the history
  • Loading branch information
Somtom committed Jan 7, 2024
1 parent b562a00 commit a38677a
Show file tree
Hide file tree
Showing 9 changed files with 152 additions and 20 deletions.
24 changes: 13 additions & 11 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,17 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]
## [Unreleased]

### Added
### Added

* Support for Redshift

### Changed

## [0.4.0]
## [0.4.0]

### Added
### Added
* You can now provide default mocks in table_meta by @Somtom in https://github.com/DeepLcom/sql-mock/pull/19
* Add array column types by @Somtom in https://github.com/DeepLcom/sql-mock/pull/22
* Use sqlglot for table ref replace by @Somtom in https://github.com/DeepLcom/sql-mock/pull/24
Expand All @@ -31,31 +33,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
* Fixed quickstart docs by @Somtom in https://github.com/DeepLcom/sql-mock/pull/20


## [0.3.1]
## [0.3.1]

### Fixed
### Fixed
- MockTable classes now have a `_sql_dialect` attribute that is used with `sglglot` for more reliable dialect conversions


## [0.3.0]
## [0.3.0]

**Full Changelog**: https://github.com/DeepLcom/sql-mock/compare/v0.2.0...v0.3.0

### Added
### Added
* Now you can also pass a `query` to the `table_meta`. The `query_path` will overwrite a `query` in case both are provided
* New method `assert_cte_equal` that allows to check the output of a specific CTE in the query you want to test.
* Added documentation page
* Added documentation page

### Changed

* The `_get_results` method now accepts a `query` instead of needing to load the query within the method. In case you created custom Mock Table classes, you need to take this into account.


## [0.2.0]
## [0.2.0]

**Full Changelog**: https://github.com/DeepLcom/sql-mock/compare/v0.1.2...v0.2.0

### Added
### Added

* Possibility to pass the `query_path` in the `table_meta` decorator. This allows to specify the query of a model in a single place instead of always needing to pass it in the `from_mocks` method. The provided query can still be overwritten in `from_mocks` if necessary. @Somtom in https://github.com/DeepLcom/sql-mock/pull/11
* Improved readme and separated contribution guidelines by @Somtom in https://github.com/DeepLcom/sql-mock/pull/9
Expand Down
8 changes: 6 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,10 @@ The primary purpose of this library is to simplify the testing of SQL data model

A full documentation can be found [on the documentation page](https://deeplcom.github.io/sql-mock/)

The library currently supports the following databases.
The library currently supports the following databases.
* BigQuery
* Clickhouse
* Redshift

## Installation

Expand All @@ -26,6 +27,9 @@ pip install --upgrade "sql-mock[bigquery]"

# Clickhouse
pip install --upgrade "sql-mock[clickhouse]"

# Redshift
pip install --upgrade "sql-mock[redshift]"
```

If you need to modify this source code, install the dependencies using poetry:
Expand Down Expand Up @@ -66,7 +70,7 @@ Find more about contributing code in the [Contribution Guidelines](./CONTRIBUTIO
### SQL Mock Buddy - A custom (Chat) GPT to support you

We ran a small experiment to create a custom GPT for SQL Mock.
The SQL Mock Buddy can be accessed here: [https://chat.openai.com/g/g-FIXNcqu1l-sql-mock-buddy](https://chat.openai.com/g/g-FIXNcqu1l-sql-mock-buddy)
The SQL Mock Buddy can be accessed here: [https://chat.openai.com/g/g-FIXNcqu1l-sql-mock-buddy](https://chat.openai.com/g/g-FIXNcqu1l-sql-mock-buddy)

SQL Mock Buddy should help you to get started quickly with SQL Mock.

Expand Down
8 changes: 4 additions & 4 deletions docsource/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

We are planning to add more and more supported database systems. However, if your system is not supported yet, you can still use SQL Mock. There are only 2 things you need to do:

### Create your `MockTable` class
### Create your `MockTable` class

First, you need to create a `MockTable` class for your database system that inherits from `sql_mock.table_mocks.BaseMockTable`.

Expand All @@ -27,7 +27,7 @@ from sql_mock.column_mocks import ColumnMock

class MyFanceDatabaseColumnMock(ColumnMock):
# In case you need some specific logic that overwrites the default behavior, you can do so here
pass
pass

class Int(MyFanceDatabaseColumnMock):
dtype = "Integer"
Expand All @@ -44,13 +44,13 @@ Feel free to create a PR on this repository that we can start supporting your da

## I am missing a specific ColumnMock type for my model fields

We implementd some basic column types but it could happen that you don't find the one you need.
We implemented some basic column types but it could happen that you don't find the one you need.
Luckily, you can easily create those with the tools provided.
The only thing you need to do is to inherit from the `ColumnMock` that is specific to your database system (e.g. `BigQueryColumnMock`) and write classes for the column mocks you are missing. Usually you only need to set the correct `dtype`. This would later be used in the `cast(col to <dtype>)` expression.

```python
# Replace the import with the database system you are using
from sql_mock.bigquery.column_mock import BigQueryColumnMock
from sql_mock.bigquery.column_mock import BigQueryColumnMock

class MyFancyMissingColType(BigQueryColumnMock):
dtype = "FancyMissingColType"
Expand Down
3 changes: 3 additions & 0 deletions docsource/getting_started/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,9 @@ pip install --upgrade "sql-mock[bigquery]"

# Clickhouse
pip install --upgrade "sql-mock[clickhouse]"

# Redshift
pip install --upgrade "sql-mock[redshift]"
```

If you need to modify this source code, install the dependencies using poetry:
Expand Down
7 changes: 4 additions & 3 deletions docsource/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
Welcome to SQL Mock's documentation!
====================================

The primary purpose of this library is to simplify the testing of SQL data models and queries by allowing users to mock input data and create tests for various scenarios.
The primary purpose of this library is to simplify the testing of SQL data models and queries by allowing users to mock input data and create tests for various scenarios.
It provides a consistent and convenient way to test the execution of your query without the need to process a massive amount of data.

.. meta::
Expand All @@ -27,7 +27,7 @@ It provides a consistent and convenient way to test the execution of your query

.. toctree::
:maxdepth: 3
:caption: Basic Usage
:caption: Basic Usage

usage/defining_table_mocks
usage/dbt
Expand All @@ -38,10 +38,11 @@ It provides a consistent and convenient way to test the execution of your query

.. toctree::
:maxdepth: 3
:caption: Database Specifics
:caption: Database Specifics

usage/bigquery/index
usage/clickhouse/index
usage/redshift/index

.. toctree::
:maxdepth: 3
Expand Down
37 changes: 37 additions & 0 deletions docsource/sql_mock.redshift.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
sql\_mock.redshift package
==========================

Submodules
----------

sql\_mock.redshift.column\_mocks module
---------------------------------------

.. automodule:: sql_mock.redshift.column_mocks
:members:
:undoc-members:
:show-inheritance:

sql\_mock.redshift.settings module
----------------------------------

.. automodule:: sql_mock.redshift.settings
:members:
:undoc-members:
:show-inheritance:

sql\_mock.redshift.table\_mocks module
--------------------------------------

.. automodule:: sql_mock.redshift.table_mocks
:members:
:undoc-members:
:show-inheritance:

Module contents
---------------

.. automodule:: sql_mock.redshift
:members:
:undoc-members:
:show-inheritance:
61 changes: 61 additions & 0 deletions docsource/usage/redshift/examples.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
```{toctree}
:maxdepth: 2
```

# Example: Testing Subscription Counts in ClickHouse

```python
import datetime
from sql_mock.redshift import column_mocks as col
from sql_mock.redshift.table_mocks import RedshiftMockTable
from sql_mock.table_mocks import table_meta

# Define mock tables for your data model that inherit from RedshiftMockTable
@table_meta(table_ref="data.users")
class UserTable(RedshiftMockTable):
user_id = col.INTEGER(default=1)
user_name = col.VARCHAR(default="Mr. T")


@table_meta(table_ref="data.subscriptions")
class SubscriptionTable(RedshiftMockTable):
subscription_id = col.INTEGER(default=1)
period_start_date = col.DATE(default=datetime.date(2023, 9, 5))
period_end_date = col.DATE(default=datetime.date(2023, 9, 5))
user_id = col.INTEGER(default=1)

# Define a mock table for your expected results
class SubscriptionCountTable(RedshiftMockTable):
subscription_count = col.INTEGER(default=1)
user_id = col.INTEGER(default=1)

# Your original SQL query
query = """
SELECT
count(*) AS subscription_count,
user_id
FROM data.users
LEFT JOIN data.subscriptions USING(user_id)
GROUP BY user_id
"""

# Create mock data for the 'data.users' and 'data.subscriptions' tables
users = UserTable.from_dicts([{'user_id': 1}, {'user_id': 2}])
subscriptions = SubscriptionTable.from_dicts([
{'subscription_id': 1, 'user_id': 1},
{'subscription_id': 2, 'user_id': 1},
{'subscription_id': 2, 'user_id': 2},
])

# Define your expected results
expected = [
{'user_id': 1, 'subscription_count': 2},
{'user_id': 2, 'subscription_count': 1}
]

# Simulate the SQL query using SQL Mock
res = SubscriptionCountTable.from_mocks(query=query, input_data=[users, subscriptions])

# Assert the results
res.assert_equal(expected)
```
10 changes: 10 additions & 0 deletions docsource/usage/redshift/index.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
Redshift
=====================

This section documents the specifics on how to use SQL Mock with Redshift

.. toctree::
:maxdepth: 4

./settings.md
./examples.md
14 changes: 14 additions & 0 deletions docsource/usage/redshift/settings.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
```{toctree}
:maxdepth: 2
```

# Settings

In order to use SQL Mock with Redshift, you need to provide the following environment variables when you run tests:

* `SQL_MOCK_REDSHIFT_HOST`: The host of your Redshift instance
* `SQL_MOCK_REDSHIFT_USER`: The user of your Redshift instance
* `SQL_MOCK_REDSHIFT_PASSWORD`: The password of your Redshift instance
* `SQL_MOCK_REDSHIFT_PORT`: The port of your Redshift instance

Having those environment variables enables SQL Mock to connect to your Redshift instance.

0 comments on commit a38677a

Please sign in to comment.