updates:
- [github.com/psf/black: 23.7.0 → 23.9.1](psf/black@23.7.0...23.9.1)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Vijay Kiran <[email protected]>

* Initial commit with csv tests

* Added Json tests

* Added support for csv, json and parquet file for
duckdb.
Changes to be committed:
    modified:   soda/duckdb/soda/data_sources/duckdb_data_source.py
	modified:   soda/duckdb/tests/test_duckdb.py

* Removing unused import
Changes to be committed:
	modified:   soda/duckdb/tests/test_duckdb.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update soda/duckdb/tests/test_duckdb.py

Removed unused f-string

Co-authored-by: Milan Lukac <[email protected]>

* Update soda/duckdb/tests/test_duckdb.py

Removed unused f-string

Co-authored-by: Milan Lukac <[email protected]>

* Update soda/duckdb/tests/test_duckdb.py

Removed unused f-string

Co-authored-by: Milan Lukac <[email protected]>

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Milan Lukac <[email protected]>

Co-authored-by: data-fool <[email protected]>

* fix: compute value counts in DB rather than in python for categoric distribution checks (#116)

* get value counts from db for categoric variables

* add/modify unit tests for distro checks

* fix unit tests

* fix sqlserver limit query

* fix db specific unit tests for distro check

* fix unit tests and change pr feedback

* do not evaluate the check if there are more than 1M groups

* fix sqlserver

* remove sampling from categoric variables for distro checks

* add query logs for distro check

* use logs.debug to show the running query in verbose mode rather than using logs.info

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

…ta#1951)

* feat: Add Dask/Pandas data source naming support

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add missing method

* fix indent

* Fix test

* remove unused feature test

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add SparkDF support

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Freshness: support mixed thresholds

* fix precommit

…a#1963)

Co-authored-by: Milan Lukac <[email protected]>

…odadata#1959)

* Sqlserver: support quoting tables with whitespace

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Support [] as quotes

* Fix parser

* Use upper case info schema

* Remove default casify sql function

* Update quoting

* Fix quoting

* Strip quotes from all payloads using table name

* Change how bracket quotes work

* Remove schema identifier lowercasing

* Fix parser

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Initial contracts transformation proposal

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Added schema check capability matching columns

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fixing the schema mismatching columns test

* Improved test suite and initial docs

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Added contract docs and missing check configurations

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Added reference property to contracts

* Added extra comment in programmatic docs and added test to ignore other keys

* added setup.py

* Preparations for contracts release

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* updated contracts package version to 3.0.54

* Fix imports

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix CI

* Fix CI

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Milan Lukac <[email protected]>

…unts. (sodadata#1973)

Co-authored-by: Milan Lukac <[email protected]>

* upgrade pydantic from v1 to v2

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Milan Lukac <[email protected]>

updates:
- [github.com/pre-commit/pre-commit-hooks: v4.4.0 → v4.5.0](pre-commit/pre-commit-hooks@v4.4.0...v4.5.0)
- [github.com/asottile/pyupgrade: v3.10.1 → v3.15.0](asottile/pyupgrade@v3.10.1...v3.15.0)
- [github.com/PyCQA/isort: 5.12.0 → 5.13.1](PyCQA/isort@5.12.0...5.13.1)
- [github.com/psf/black: 23.9.1 → 23.11.0](psf/black@23.9.1...23.11.0)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

…1980)

* feat: implement warn_only for anomaly score (sodadata#156)

* implement warn_only for anomaly score

* fix unit tests

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

…ata#1984)

* Added group by example for Soda Core with failed rows check

* Adjusted SQL per suggestion

* Corrected SQL

…1994)

* bug: handle null values for continuous dist (sodadata#165)

* handle null values for ks test

* remove null values from test data

* add unit tests for partially null columns

* fix unit tests for empty test set

* raise warning if we remove null values

* refac warning string

* add unit tests for null values in dro generation

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add future annotations

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

updates:
- [github.com/PyCQA/isort: 5.13.1 → 5.13.2](PyCQA/isort@5.13.1...5.13.2)
- [github.com/psf/black: 23.11.0 → 23.12.1](psf/black@23.11.0...23.12.1)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Milan Lukac <[email protected]>

* Merge branch 'main' of https://github.com/sodadata/soda-core

* run antlr parser for core

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix conflicts and fix python3.8 issues

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

support built-in prophet public holidays - cherry-picked from library

* feat: Increase PyHive Log Level to ERROR

* fix: DESCRIBE without TABLE qualifier as it does not exists in Hive and works in Spark without it

* feat: Add kerberos service property to allow kerberized hive connection

---------

Co-authored-by: Milan Lukac <[email protected]>

…ata#1998)

* Implement migrate from anomaly score check config

* rename migrate_anomaly_score to take_over_old_check

* rename config to take_over_existing_anomaly_score_check

* cherry-pick from library

* fix dogfooding feedbacks (#175)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* trigger CI

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/psf/black: 23.12.1 → 24.1.1](psf/black@23.12.1...24.1.1)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* feat: anomaly detection simulator (#163)

Co-authored-by: Milan Lukac <[email protected]>

…odadata#2013)

* feat: correctly identified anomalies are excluded from training data (sodadata#178)

* correctly identified anomalies are excluded from training data

* fix in case skipMeasurements does not exist

* fix unit tests

* Fix Bastiens feedback

* remove a misleading comment

* use isCorrectlyClassified instead of isAnomaly field

* rename is_anomaly fields to increase the readibility

* fix formatting

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

…irst (sodadata#2014)

* fix: show more clearly the detected frequency using warning message first (sodadata#180)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Fix simulator streamlit path

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix simulator streamlit path

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

updates:
- [github.com/psf/black: 24.1.1 → 24.2.0](psf/black@24.1.1...24.2.0)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Update oracle_data_source.py

Accommodate the Oracle data source to connect with `host`, `port` and `service_name` passed to it as different properties in the soda config file.
We can still pass `connectstring` property.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update oracle_data_source.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* this commit solves sodadata#1990

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Milan Lukac <[email protected]>

)

…ults endpoint instead of test results (sodadata#2025)

* fix timezone issue and log messages (#188)

* feat: in anomaly detection simulator use soda core historic check results endpoint instead of test results (#190)

* Fix integer detection

* Use sodaCoreCheckIdentities endpoint to get check ids for simulator

* Unpin dask-sql

* Unpin dask-sql

* Fix custom functions

…sodadata#2027)

* IA 533 implement daily and monthly seasonality to external regressor (#189)

* implement weekly montly yearly seasonality for ext regressor

* Implement custom external regressor for monthly and yearly

* Fix integer detection

* Use sodaCoreCheckIdentities endpoint to get check ids for simulator

* refactor external regressor and add monthly and yearly feedbacks

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

updates:
- [github.com/asottile/pyupgrade: v3.15.0 → v3.15.1](asottile/pyupgrade@v3.15.0...v3.15.1)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Milan Lukac <[email protected]>

* initial commit with regression

* possible regression fix

* build fix

* use new dbt access url pattern to reach cloud API

* add some spaces

* added contract roadmap ideas

* Added column level checks and switched to standard yaml checks

* Added contract json schema and started on docs

* Contract wip

* Contract wip

* Contract wip

* Contract wip

* Contract wip

* Contract wip

* Contract wip

* Contract wip

* Contract wip

* Contract wip

* Contract wip

* Contract wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* wip

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* wip

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* wip

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* wip

* Update soda/contracts/docs/README.md

Co-authored-by: Milan Lukac <[email protected]>

* Update soda/contracts/docs/README.md

Co-authored-by: Milan Lukac <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* wip

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* wip

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* wip

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* wip

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* wip

* wip

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* wip

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* wip

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* wip

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* wip

* wip

* wip

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* wip

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* wip

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* wip

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* wip

* wip

* wip

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* wip

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* wip

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix import path in tests

* wip

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* wip

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* wip

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* wip

* wip

* Fix attribute handler timezone test

* Fix fixtures imports

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix fixtures imports

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Milan Lukac <[email protected]>
Co-authored-by: Milan Lukac <[email protected]>

…2038)

The argument name 'set_scan_results_file' in the set_scan_results_file
method was confusing as it shared the same name as the function itself.
This commit renames the argument to 'scan_results_file' for improved
clarity and consistency. Resolves sodadata#2040

updates:
- [github.com/pre-commit/pre-commit-hooks: v4.5.0 → v4.6.0](pre-commit/pre-commit-hooks@v4.5.0...v4.6.0)
- [github.com/asottile/pyupgrade: v3.15.1 → v3.15.2](asottile/pyupgrade@v3.15.1...v3.15.2)
- [github.com/psf/black: 24.2.0 → 24.4.0](psf/black@24.2.0...24.4.0)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Update db2_data_source.py

This is needed in order to fix the connectivity issues for db2 from soda-core

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update db2_data_source.py

adding security as part of the config for db2, security will be added to the connection string only when it is not empty. This is needed in order to establish a connection with db2

* Update db2_data_source.py

remove security from the 1st connection string

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Milan Lukac <[email protected]>