Sync main from soda-core #9
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
updates: - [github.com/psf/black: 23.7.0 → 23.9.1](psf/black@23.7.0...23.9.1) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Vijay Kiran <[email protected]>
* Initial commit with csv tests
* Added Json tests
* Added support for csv, json and parquet file for
duckdb.
Changes to be committed:
modified: soda/duckdb/soda/data_sources/duckdb_data_source.py
modified: soda/duckdb/tests/test_duckdb.py
* Removing unused import
Changes to be committed:
modified: soda/duckdb/tests/test_duckdb.py
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update soda/duckdb/tests/test_duckdb.py
Removed unused f-string
Co-authored-by: Milan Lukac <[email protected]>
* Update soda/duckdb/tests/test_duckdb.py
Removed unused f-string
Co-authored-by: Milan Lukac <[email protected]>
* Update soda/duckdb/tests/test_duckdb.py
Removed unused f-string
Co-authored-by: Milan Lukac <[email protected]>
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Milan Lukac <[email protected]>
Co-authored-by: data-fool <[email protected]>
* fix: compute value counts in DB rather than in python for categoric distribution checks (#116) * get value counts from db for categoric variables * add/modify unit tests for distro checks * fix unit tests * fix sqlserver limit query * fix db specific unit tests for distro check * fix unit tests and change pr feedback * do not evaluate the check if there are more than 1M groups * fix sqlserver * remove sampling from categoric variables for distro checks * add query logs for distro check * use logs.debug to show the running query in verbose mode rather than using logs.info * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
…ta#1951) * feat: Add Dask/Pandas data source naming support * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add missing method * fix indent * Fix test * remove unused feature test * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add SparkDF support * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Freshness: support mixed thresholds * fix precommit
Co-authored-by: Milan Lukac <[email protected]>
…odadata#1959) * Sqlserver: support quoting tables with whitespace * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Support [] as quotes * Fix parser * Use upper case info schema * Remove default casify sql function * Update quoting * Fix quoting * Strip quotes from all payloads using table name * Change how bracket quotes work * Remove schema identifier lowercasing * Fix parser --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Initial contracts transformation proposal * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Added schema check capability matching columns * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixing the schema mismatching columns test * Improved test suite and initial docs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Added contract docs and missing check configurations * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Added reference property to contracts * Added extra comment in programmatic docs and added test to ignore other keys * added setup.py * Preparations for contracts release * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * updated contracts package version to 3.0.54 * Fix imports * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix CI * Fix CI --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Milan Lukac <[email protected]>
The argument name 'set_scan_results_file' in the set_scan_results_file method was confusing as it shared the same name as the function itself. This commit renames the argument to 'scan_results_file' for improved clarity and consistency. Resolves sodadata#2040
updates: - [github.com/pre-commit/pre-commit-hooks: v4.5.0 → v4.6.0](pre-commit/pre-commit-hooks@v4.5.0...v4.6.0) - [github.com/asottile/pyupgrade: v3.15.1 → v3.15.2](asottile/pyupgrade@v3.15.1...v3.15.2) - [github.com/psf/black: 24.2.0 → 24.4.0](psf/black@24.2.0...24.4.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Update db2_data_source.py This is needed in order to fix the connectivity issues for db2 from soda-core * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update db2_data_source.py adding security as part of the config for db2, security will be added to the connection string only when it is not empty. This is needed in order to establish a connection with db2 * Update db2_data_source.py remove security from the 1st connection string * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Milan Lukac <[email protected]>
Contracts v3: * Added spark session support * Made all sql properties consistently end with ..._sql * Introduced warehouse as terminology * Made identity hash instead of long structured name * Added support quoting * Added check level filter_sql support
* Bump dev requirements * Downgrade docutil for py38 compatibility
* User defined metric check: support failed rows query * Test file version as well * Fix CI * Make metric check cfg contructor flexy * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Improve memory usage * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix broken change * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add pre-commit * Format correctly, add missing log methods * Fix tests * Use correct Query class * Fix pyspark version * dev requirements conflict --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Bump requests and tox/docker * Bump requests and tox/docker
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.