From a38677a3445a2a7a4c81cb6f55cbe696cd5816b7 Mon Sep 17 00:00:00 2001 From: Thomas Schmidt Date: Sun, 7 Jan 2024 14:10:06 +0100 Subject: [PATCH] Added Redshift docs --- CHANGELOG.md | 24 +++++---- README.md | 8 ++- docsource/faq.md | 8 +-- docsource/getting_started/installation.md | 3 ++ docsource/index.rst | 7 +-- docsource/sql_mock.redshift.rst | 37 ++++++++++++++ docsource/usage/redshift/examples.md | 61 +++++++++++++++++++++++ docsource/usage/redshift/index.rst | 10 ++++ docsource/usage/redshift/settings.md | 14 ++++++ 9 files changed, 152 insertions(+), 20 deletions(-) create mode 100644 docsource/sql_mock.redshift.rst create mode 100644 docsource/usage/redshift/examples.md create mode 100644 docsource/usage/redshift/index.rst create mode 100644 docsource/usage/redshift/settings.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 4bef76c..80f21f5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,15 +9,17 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [Unreleased] +## [Unreleased] -### Added +### Added + +* Support for Redshift ### Changed -## [0.4.0] +## [0.4.0] -### Added +### Added * You can now provide default mocks in table_meta by @Somtom in https://github.com/DeepLcom/sql-mock/pull/19 * Add array column types by @Somtom in https://github.com/DeepLcom/sql-mock/pull/22 * Use sqlglot for table ref replace by @Somtom in https://github.com/DeepLcom/sql-mock/pull/24 @@ -31,31 +33,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 * Fixed quickstart docs by @Somtom in https://github.com/DeepLcom/sql-mock/pull/20 -## [0.3.1] +## [0.3.1] -### Fixed +### Fixed - MockTable classes now have a `_sql_dialect` attribute that is used with `sglglot` for more reliable dialect conversions -## [0.3.0] +## [0.3.0] **Full Changelog**: https://github.com/DeepLcom/sql-mock/compare/v0.2.0...v0.3.0 -### Added +### Added * Now you can also pass a `query` to the `table_meta`. The `query_path` will overwrite a `query` in case both are provided * New method `assert_cte_equal` that allows to check the output of a specific CTE in the query you want to test. -* Added documentation page +* Added documentation page ### Changed * The `_get_results` method now accepts a `query` instead of needing to load the query within the method. In case you created custom Mock Table classes, you need to take this into account. -## [0.2.0] +## [0.2.0] **Full Changelog**: https://github.com/DeepLcom/sql-mock/compare/v0.1.2...v0.2.0 -### Added +### Added * Possibility to pass the `query_path` in the `table_meta` decorator. This allows to specify the query of a model in a single place instead of always needing to pass it in the `from_mocks` method. The provided query can still be overwritten in `from_mocks` if necessary. @Somtom in https://github.com/DeepLcom/sql-mock/pull/11 * Improved readme and separated contribution guidelines by @Somtom in https://github.com/DeepLcom/sql-mock/pull/9 diff --git a/README.md b/README.md index 61c917d..38e7a9f 100644 --- a/README.md +++ b/README.md @@ -12,9 +12,10 @@ The primary purpose of this library is to simplify the testing of SQL data model A full documentation can be found [on the documentation page](https://deeplcom.github.io/sql-mock/) -The library currently supports the following databases. +The library currently supports the following databases. * BigQuery * Clickhouse +* Redshift ## Installation @@ -26,6 +27,9 @@ pip install --upgrade "sql-mock[bigquery]" # Clickhouse pip install --upgrade "sql-mock[clickhouse]" + +# Redshift +pip install --upgrade "sql-mock[redshift]" ``` If you need to modify this source code, install the dependencies using poetry: @@ -66,7 +70,7 @@ Find more about contributing code in the [Contribution Guidelines](./CONTRIBUTIO ### SQL Mock Buddy - A custom (Chat) GPT to support you We ran a small experiment to create a custom GPT for SQL Mock. -The SQL Mock Buddy can be accessed here: [https://chat.openai.com/g/g-FIXNcqu1l-sql-mock-buddy](https://chat.openai.com/g/g-FIXNcqu1l-sql-mock-buddy) +The SQL Mock Buddy can be accessed here: [https://chat.openai.com/g/g-FIXNcqu1l-sql-mock-buddy](https://chat.openai.com/g/g-FIXNcqu1l-sql-mock-buddy) SQL Mock Buddy should help you to get started quickly with SQL Mock. diff --git a/docsource/faq.md b/docsource/faq.md index 59dd349..095a1a0 100644 --- a/docsource/faq.md +++ b/docsource/faq.md @@ -5,7 +5,7 @@ We are planning to add more and more supported database systems. However, if your system is not supported yet, you can still use SQL Mock. There are only 2 things you need to do: -### Create your `MockTable` class +### Create your `MockTable` class First, you need to create a `MockTable` class for your database system that inherits from `sql_mock.table_mocks.BaseMockTable`. @@ -27,7 +27,7 @@ from sql_mock.column_mocks import ColumnMock class MyFanceDatabaseColumnMock(ColumnMock): # In case you need some specific logic that overwrites the default behavior, you can do so here - pass + pass class Int(MyFanceDatabaseColumnMock): dtype = "Integer" @@ -44,13 +44,13 @@ Feel free to create a PR on this repository that we can start supporting your da ## I am missing a specific ColumnMock type for my model fields -We implementd some basic column types but it could happen that you don't find the one you need. +We implemented some basic column types but it could happen that you don't find the one you need. Luckily, you can easily create those with the tools provided. The only thing you need to do is to inherit from the `ColumnMock` that is specific to your database system (e.g. `BigQueryColumnMock`) and write classes for the column mocks you are missing. Usually you only need to set the correct `dtype`. This would later be used in the `cast(col to )` expression. ```python # Replace the import with the database system you are using -from sql_mock.bigquery.column_mock import BigQueryColumnMock +from sql_mock.bigquery.column_mock import BigQueryColumnMock class MyFancyMissingColType(BigQueryColumnMock): dtype = "FancyMissingColType" diff --git a/docsource/getting_started/installation.md b/docsource/getting_started/installation.md index baa1fba..d80e152 100644 --- a/docsource/getting_started/installation.md +++ b/docsource/getting_started/installation.md @@ -12,6 +12,9 @@ pip install --upgrade "sql-mock[bigquery]" # Clickhouse pip install --upgrade "sql-mock[clickhouse]" + +# Redshift +pip install --upgrade "sql-mock[redshift]" ``` If you need to modify this source code, install the dependencies using poetry: diff --git a/docsource/index.rst b/docsource/index.rst index 2f84192..98e1a65 100644 --- a/docsource/index.rst +++ b/docsource/index.rst @@ -6,7 +6,7 @@ Welcome to SQL Mock's documentation! ==================================== -The primary purpose of this library is to simplify the testing of SQL data models and queries by allowing users to mock input data and create tests for various scenarios. +The primary purpose of this library is to simplify the testing of SQL data models and queries by allowing users to mock input data and create tests for various scenarios. It provides a consistent and convenient way to test the execution of your query without the need to process a massive amount of data. .. meta:: @@ -27,7 +27,7 @@ It provides a consistent and convenient way to test the execution of your query .. toctree:: :maxdepth: 3 - :caption: Basic Usage + :caption: Basic Usage usage/defining_table_mocks usage/dbt @@ -38,10 +38,11 @@ It provides a consistent and convenient way to test the execution of your query .. toctree:: :maxdepth: 3 - :caption: Database Specifics + :caption: Database Specifics usage/bigquery/index usage/clickhouse/index + usage/redshift/index .. toctree:: :maxdepth: 3 diff --git a/docsource/sql_mock.redshift.rst b/docsource/sql_mock.redshift.rst new file mode 100644 index 0000000..1a3a286 --- /dev/null +++ b/docsource/sql_mock.redshift.rst @@ -0,0 +1,37 @@ +sql\_mock.redshift package +========================== + +Submodules +---------- + +sql\_mock.redshift.column\_mocks module +--------------------------------------- + +.. automodule:: sql_mock.redshift.column_mocks + :members: + :undoc-members: + :show-inheritance: + +sql\_mock.redshift.settings module +---------------------------------- + +.. automodule:: sql_mock.redshift.settings + :members: + :undoc-members: + :show-inheritance: + +sql\_mock.redshift.table\_mocks module +-------------------------------------- + +.. automodule:: sql_mock.redshift.table_mocks + :members: + :undoc-members: + :show-inheritance: + +Module contents +--------------- + +.. automodule:: sql_mock.redshift + :members: + :undoc-members: + :show-inheritance: diff --git a/docsource/usage/redshift/examples.md b/docsource/usage/redshift/examples.md new file mode 100644 index 0000000..f736773 --- /dev/null +++ b/docsource/usage/redshift/examples.md @@ -0,0 +1,61 @@ +```{toctree} +:maxdepth: 2 +``` + +# Example: Testing Subscription Counts in ClickHouse + +```python +import datetime +from sql_mock.redshift import column_mocks as col +from sql_mock.redshift.table_mocks import RedshiftMockTable +from sql_mock.table_mocks import table_meta + +# Define mock tables for your data model that inherit from RedshiftMockTable +@table_meta(table_ref="data.users") +class UserTable(RedshiftMockTable): + user_id = col.INTEGER(default=1) + user_name = col.VARCHAR(default="Mr. T") + + +@table_meta(table_ref="data.subscriptions") +class SubscriptionTable(RedshiftMockTable): + subscription_id = col.INTEGER(default=1) + period_start_date = col.DATE(default=datetime.date(2023, 9, 5)) + period_end_date = col.DATE(default=datetime.date(2023, 9, 5)) + user_id = col.INTEGER(default=1) + +# Define a mock table for your expected results +class SubscriptionCountTable(RedshiftMockTable): + subscription_count = col.INTEGER(default=1) + user_id = col.INTEGER(default=1) + +# Your original SQL query +query = """ +SELECT + count(*) AS subscription_count, + user_id +FROM data.users +LEFT JOIN data.subscriptions USING(user_id) +GROUP BY user_id +""" + +# Create mock data for the 'data.users' and 'data.subscriptions' tables +users = UserTable.from_dicts([{'user_id': 1}, {'user_id': 2}]) +subscriptions = SubscriptionTable.from_dicts([ + {'subscription_id': 1, 'user_id': 1}, + {'subscription_id': 2, 'user_id': 1}, + {'subscription_id': 2, 'user_id': 2}, +]) + +# Define your expected results +expected = [ + {'user_id': 1, 'subscription_count': 2}, + {'user_id': 2, 'subscription_count': 1} +] + +# Simulate the SQL query using SQL Mock +res = SubscriptionCountTable.from_mocks(query=query, input_data=[users, subscriptions]) + +# Assert the results +res.assert_equal(expected) +``` diff --git a/docsource/usage/redshift/index.rst b/docsource/usage/redshift/index.rst new file mode 100644 index 0000000..b2f7caa --- /dev/null +++ b/docsource/usage/redshift/index.rst @@ -0,0 +1,10 @@ +Redshift +===================== + +This section documents the specifics on how to use SQL Mock with Redshift + +.. toctree:: + :maxdepth: 4 + + ./settings.md + ./examples.md diff --git a/docsource/usage/redshift/settings.md b/docsource/usage/redshift/settings.md new file mode 100644 index 0000000..fdf1ba0 --- /dev/null +++ b/docsource/usage/redshift/settings.md @@ -0,0 +1,14 @@ +```{toctree} +:maxdepth: 2 +``` + +# Settings + +In order to use SQL Mock with Redshift, you need to provide the following environment variables when you run tests: + +* `SQL_MOCK_REDSHIFT_HOST`: The host of your Redshift instance +* `SQL_MOCK_REDSHIFT_USER`: The user of your Redshift instance +* `SQL_MOCK_REDSHIFT_PASSWORD`: The password of your Redshift instance +* `SQL_MOCK_REDSHIFT_PORT`: The port of your Redshift instance + +Having those environment variables enables SQL Mock to connect to your Redshift instance.