V5 #66
Draft
V5 #66
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* feat: v5 example * pylint * refactor: Change environment.py to keyed.py and change to abs imports * feat: Make inject return self and implement for keyed resource * feat: Change injection logic slightly and simplify injection code * feat: Add more tests, add InjectionError * feat: make keyed resource immutable. * feat: Make file resources leaner * feat: Remove pylint from tests
* feat: v5 example * pylint * refactor: Change environment.py to keyed.py and change to abs imports * feat: Make inject return self and implement for keyed resource * feat: Change injection logic slightly and simplify injection code * feat: Add more tests, add InjectionError * feat: Add S3 resources * docs: Add attributes * feat: make keyed resource immutable. * feat: Make file resources leaner * feat: Remove pylint from tests * fix: HDF File resource * refactor: use static methods for file handlers * feat: Make S3 file resources leaner & refactor * feat: Add test for pandera validations * fix: Tests & refactor * refactor: Parquet test * feat: Add file tests for all file-types * refactor: Only use mocked class for base file test * refactor: Clean up & simplify file tests * feat: Add basic tests for s3 files * feat: Add s3 hdf test * refactor: Improve injection * feat: Add postgres handler and tests * feat: Add kafka resource and tests * refactor: Parametrise file tests * refactor: Parametrise s3 tests * feat: Add more injections and test for postgres * refactor: Minimal * refactor: Add docstrings * docs: Xor * refactor: no_disk_space -> force_read_to_memory * feat: Add postgres schema definition to postgres resource * feat: Restructure to use config+handlers >>> no inheritance * refactor: Minor inject changes * refactor: Ignore R0801 for now * fix: Postgres write with schema, add demos * feat: Add tests for keyedresource * refactor: handlers module to io and Handlers classes to Resource
* feat: Adding '---' to the beginning of yaml files generated by cli (#48) * feat: Adding '---' to the beginning of yaml files generated by cli * feat: renaming variable * feat: add resource migration script * feat: Add migration cli * feat: add schema migration script * feat: Working migration command for schemas * Add capability to convert both schemas and resources --------- Co-authored-by: Christiano <[email protected]> Co-authored-by: Artur Krochin <[email protected]>
* feat: Adding '---' to the beginning of yaml files generated by cli (#48) * feat: Adding '---' to the beginning of yaml files generated by cli * feat: renaming variable * feat: add resource migration script * feat: Add migration cli * feat: add schema migration script * feat: Working migration command for schemas * Add capability to convert both schemas and resources * Add uhura & serde for file resources * Correct typehint * Change hdf extension * Add uhura tests & fix missing test_path attributes * Add uhura for s3 resources (& tests) * Fix migration script confirm prompt * Add injections and checking in all relevant places * Add postgres + uhura resource * Add missing check_injections to postgres * Correct postgres cache key and test * Add kafka + uhura resource * Remove dead code * Remove dead code * Add and improve tests * Fix test writing to files * Remove dead code * Format code * Add kafka inject tests * Format code * Add tests for serde and change serde init signature * Add resource test_path inject test * Update migration scripts to work for new uhura based resources * Add brackets --------- Co-authored-by: Christiano <[email protected]> Co-authored-by: Artur Krochin <[email protected]>
* feat: Adding application name option * chore: update make file and requirements
* feat: Adding '---' to the beginning of yaml files generated by cli (#48) * feat: Adding '---' to the beginning of yaml files generated by cli * feat: renaming variable * feat: add resource migration script * feat: Add migration cli * feat: add schema migration script * feat: Working migration command for schemas * Add capability to convert both schemas and resources * feat: add support for has_no_nulls validation * feat: add support for is_in validator * feat: add support for uniqueness validator * feat: add support for greater than validation * fix: fix issue with empty normalized col names * feat: add validator for greater than or equals * feat: add validator for is between * fix: remove scaffolding * fix: optimise imports * feat: add note about custom validations * feat: Add demo files * feat: Correct demo files * feat: Add less than and less equals to schema migration cli * feat: Add migration schema match_all warning --------- Co-authored-by: Christiano <[email protected]> Co-authored-by: Artur Krochin <[email protected]>
rnd 2867 metrics
Overview Refactor according after discussions with @matteo-pallini last month. I had to add the custom inject back in, rather than using .format as .format requires all variables to be formatted (could use std templates but that would change the syntax to $). Validation now happens in the attached serde.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.