-
Notifications
You must be signed in to change notification settings - Fork 5
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Thomas Schmidt
committed
Nov 3, 2023
1 parent
b256279
commit 8ac02f3
Showing
100 changed files
with
10,180 additions
and
29 deletions.
There are no files selected for viewing
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
# Sphinx build info version 1 | ||
# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. | ||
config: 430fa74d48604fdb2a6cbe0d59bef63c | ||
tags: 645f666f9bcd5a90fca523b33c5a78b7 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,62 @@ | ||
|
||
# FAQ | ||
|
||
## My database system is not supported yet but I want to use SQL Mock. What should I do? | ||
|
||
We are planning to add more and more supported database systems. However, if your system is not supported yet, you can still use SQL Mock. There are only 2 things you need to do: | ||
|
||
### Create your `MockTable` class | ||
|
||
First, you need to create a `MockTable` class for your database system that inherits from `sql_mock.table_mocks.BaseMockTable`. | ||
|
||
That class needs to implement the `_get_results` method which should make sure to fetch the results of a query (e.g. produced by `self._generate_query()`) and return it as list of dictionaries. | ||
|
||
Look at one of the existing client libraries to see how this could work (e.g. [BigQueryMockTable](https://github.com/DeepLcom/sql-mock/blob/main/src/sql_mock/bigquery/table_mocks.py)). | ||
|
||
You might want to create a settings class as well in case you need some specific connection settings to be available within the `_get_results` method. | ||
|
||
### Create your `ColumnMocks` | ||
|
||
Your database system might support specific database types. In order to make them available as column types, you can use the `sql_mock.column_mocks.ColumnMock` class as a base and inherit your specific column types from it. | ||
For most of your column mocks you might only need to specify the `dtype` that should be used to parse the inputs. | ||
|
||
A good practise is to create a `ColumnMock` class that is specific to your database and inherit all your column types from it, e.g.: | ||
|
||
```python | ||
from sql_mock.column_mocks import ColumnMock | ||
|
||
class MyFanceDatabaseColumnMock(ColumnMock): | ||
# In case you need some specific logic that overwrites the default behavior, you can do so here | ||
pass | ||
|
||
class Int(MyFanceDatabaseColumnMock): | ||
dtype = "Integer" | ||
|
||
class String(MyFanceDatabaseColumnMock): | ||
dtype = "String" | ||
``` | ||
|
||
### Contribute your database setup | ||
|
||
There will definitely be folks in the community that are in the need of support for the database you just created all the setup for. | ||
Feel free to create a PR on this repository that we can start supporting your database system! | ||
|
||
|
||
## I am missing a specific ColumnMock type for my model fields | ||
|
||
We implementd some basic column types but it could happen that you don't find the one you need. | ||
Luckily, you can easily create those with the tools provided. | ||
The only thing you need to do is to inherit from the `ColumnMock` that is specific to your database system (e.g. `BigQueryColumnMock`) and write classes for the column mocks you are missing. Usually you only need to set the correct `dtype`. This would later be used in the `cast(col to <dtype>)` expression. | ||
|
||
```python | ||
# Replace the import with the database system you are using | ||
from sql_mock.bigquery.column_mock import BigQueryColumnMock | ||
|
||
class MyFancyMissingColType(BigQueryColumnMock): | ||
dtype = "FancyMissingColType" | ||
|
||
# In case you need to implement additional logic for casting, you can do so here | ||
... | ||
``` | ||
|
||
**Don't forget to create a PR in case you feel that your column mock type could be useful for the community**! |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,33 @@ | ||
```{toctree} | ||
:maxdepth: 2 | ||
``` | ||
|
||
# Installation | ||
|
||
The library can be installed from [PyPI](https://pypi.org/project/sql-mock/) using pip: | ||
|
||
```shell | ||
# BigQuery | ||
pip install --upgrade "sql-mock[bigquery]" | ||
|
||
# Clickhouse | ||
pip install --upgrade "sql-mock[clickhouse]" | ||
``` | ||
|
||
If you need to modify this source code, install the dependencies using poetry: | ||
|
||
```shell | ||
poetry install --all-extras | ||
``` | ||
|
||
|
||
## Recommended Setup for Pytest | ||
If you are using pytest, make sure to add a `conftest.py` file to the root of your project. | ||
In the file add the following lines: | ||
```python | ||
import pytest | ||
pytest.register_assert_rewrite('sql_mock') | ||
``` | ||
This allows you to get a rich comparison when using the `.assert_equal` method on the table mock instances. | ||
|
||
We also recommend using [pytest-icdiff](https://github.com/hjwp/pytest-icdiff) for better visibility on diffs of failed tests. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,80 @@ | ||
```{toctree} | ||
:maxdepth: 2 | ||
``` | ||
|
||
# Quickstart | ||
|
||
Before diving into specific database scenarios, let's start with a simplified example of how SQL Mock works behind the scenes. | ||
|
||
|
||
1. You have an original SQL query, for instance: | ||
```sql | ||
-- path/to/query_for_result_table.sql | ||
SELECT id FROM data.table1 | ||
``` | ||
|
||
|
||
2. Using SQL Mock, you define mock tables. You can use the built-in column types provided by SQL Mock. Available column types include `Int`, `String`, `Date`, and more. Each database type has their own column types. Define your tables by subclassing a mock table class that fits your database (e.g. `BigQueryMockTable`) and specifying the column types along with default values. In our example we use the `ClickhouseTableMock` class | ||
```python | ||
from sql_mock.clickhouse import column_mocks as col | ||
from sql_mock.clickhouse.table_mocks import ClickHouseTableMock, table_meta | ||
|
||
@table_meta(table_ref='data.table1') | ||
class Table(ClickHouseTableMock): | ||
id = col.Int(default=1) | ||
name = col.String(default='Peter') | ||
|
||
@table_meta(table_ref='data.result_table', query_path='path/to/query_for_result_table.sql') | ||
class ResultTable(ClickhouseTableMock): | ||
id = col.Int(default=1) | ||
``` | ||
|
||
3. **Creating mock data:** Define mock data for your tables using dictionaries. Each dictionary represents a row in the table, with keys corresponding to column names. Table column keys that don't get a value will use the default. | ||
```python | ||
user_data = [ | ||
{}, # This will use the defaults for both id and name | ||
{'id': 2, 'name': 'Martin'}, | ||
{'id': 3}, # This will use defaults for the name | ||
] | ||
|
||
input_table_mock = Table.from_dicts(user_data) | ||
``` | ||
|
||
|
||
4. **Getting results for a table mock:** Use the `from_mocks` method of the table mock object to generate mock query results based on your mock data. | ||
```python | ||
res = ResultTable.from_mocks(input_data=[input_table_mock]) | ||
``` | ||
|
||
5. Behind the scene SQL Mock replaces table references (e.g. `data.table1`) in your query with Common Table Expressions (CTEs) filled with dummy data. It can roughly be compared to something like this: | ||
```sql | ||
WITH data__table1 AS ( | ||
-- Mocked inputs | ||
SELECT | ||
cast('1' AS 'String') AS id, | ||
cast('Peter' AS 'String') AS name | ||
UNION ALL | ||
SELECT | ||
cast('2' AS 'String') AS id, | ||
cast('Martin' AS 'String') AS name | ||
UNION ALL | ||
SELECT | ||
cast('3' AS 'String') AS id, | ||
cast('Peter' AS 'String') AS name | ||
) | ||
|
||
result AS ( | ||
-- Original query with replaced references | ||
SELECT id FROM data__table1 | ||
) | ||
|
||
SELECT | ||
cast(id AS 'String') AS id | ||
FROM result | ||
``` | ||
|
||
6. Finally, you can compare your results to some expected results using the `assert_equal` method. | ||
```python | ||
expected = [{'id': '1'},{'id': '2'},{'id': '3'}] | ||
res.assert_equal(expected) | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,56 @@ | ||
.. SQL Mock documentation master file, created by | ||
sphinx-quickstart on Wed Nov 1 07:36:03 2023. | ||
You can adapt this file completely to your liking, but it should at least | ||
contain the root `toctree` directive. | ||
Welcome to SQL Mock's documentation! | ||
==================================== | ||
|
||
The primary purpose of this library is to simplify the testing of SQL data models and queries by allowing users to mock input data and create tests for various scenarios. | ||
It provides a consistent and convenient way to test the execution of your query without the need to process a massive amount of data. | ||
|
||
.. meta:: | ||
:google-site-verification: ohDOHPn1YMLYMaWQpFmqQfPW_KRZXNK9Gq47pP57VQM | ||
|
||
.. automodule:: sql_mock | ||
:members: | ||
:undoc-members: | ||
:show-inheritance: | ||
|
||
.. toctree:: | ||
:maxdepth: 3 | ||
:caption: Getting Started | ||
|
||
getting_started/installation | ||
getting_started/quickstart | ||
faq | ||
|
||
.. toctree:: | ||
:maxdepth: 3 | ||
:caption: Basic Usage | ||
|
||
usage/defining_table_mocks | ||
usage/your_sql_query_to_test | ||
usage/result_assertion | ||
usage/examples | ||
|
||
.. toctree:: | ||
:maxdepth: 3 | ||
:caption: Database Specifics | ||
|
||
usage/bigquery/index | ||
usage/clickhouse/index | ||
|
||
.. toctree:: | ||
:maxdepth: 3 | ||
:caption: API Reference | ||
|
||
API Reference <modules> | ||
|
||
|
||
Indices and tables | ||
================== | ||
|
||
* :ref:`genindex` | ||
* :ref:`modindex` | ||
* :ref:`search` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
sql_mock | ||
======== | ||
|
||
.. toctree:: | ||
:maxdepth: 4 | ||
:hidden: | ||
|
||
sql_mock |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,37 @@ | ||
sql\_mock.bigquery package | ||
========================== | ||
|
||
Submodules | ||
---------- | ||
|
||
sql\_mock.bigquery.column\_mocks module | ||
--------------------------------------- | ||
|
||
.. automodule:: sql_mock.bigquery.column_mocks | ||
:members: | ||
:undoc-members: | ||
:show-inheritance: | ||
|
||
sql\_mock.bigquery.settings module | ||
---------------------------------- | ||
|
||
.. automodule:: sql_mock.bigquery.settings | ||
:members: | ||
:undoc-members: | ||
:show-inheritance: | ||
|
||
sql\_mock.bigquery.table\_mocks module | ||
-------------------------------------- | ||
|
||
.. automodule:: sql_mock.bigquery.table_mocks | ||
:members: | ||
:undoc-members: | ||
:show-inheritance: | ||
|
||
Module contents | ||
--------------- | ||
|
||
.. automodule:: sql_mock.bigquery | ||
:members: | ||
:undoc-members: | ||
:show-inheritance: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,37 @@ | ||
sql\_mock.clickhouse package | ||
============================ | ||
|
||
Submodules | ||
---------- | ||
|
||
sql\_mock.clickhouse.column\_mocks module | ||
----------------------------------------- | ||
|
||
.. automodule:: sql_mock.clickhouse.column_mocks | ||
:members: | ||
:undoc-members: | ||
:show-inheritance: | ||
|
||
sql\_mock.clickhouse.settings module | ||
------------------------------------ | ||
|
||
.. automodule:: sql_mock.clickhouse.settings | ||
:members: | ||
:undoc-members: | ||
:show-inheritance: | ||
|
||
sql\_mock.clickhouse.table\_mocks module | ||
---------------------------------------- | ||
|
||
.. automodule:: sql_mock.clickhouse.table_mocks | ||
:members: | ||
:undoc-members: | ||
:show-inheritance: | ||
|
||
Module contents | ||
--------------- | ||
|
||
.. automodule:: sql_mock.clickhouse | ||
:members: | ||
:undoc-members: | ||
:show-inheritance: |
Oops, something went wrong.