Fc into earthscope #286
Merged
Fc into earthscope #286
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add unit test to cover basic creation of fcs for synthetic data - add core FC building methods to pipelines - modify make_mth5_from_ascii so that - main takes version as a kwarg - ensure all retuned mth5_paths are instances of pathlib.Path Relates to issue #278
- utilize save_fcs and save_fcs_type in updated processing config - deprecate save_fcs kwarg from process_mth5 - minor tidying
- uncomment code to add FC layer in process_mth5 - rearrange some notes/comments in TFK - also, test_doc failing, 3.9+, 3.8 OK,
- add check to fc metadata - change processing_summary validation to use the value in metadata as this property was added to mt_metadata in aurora processing config
…info into h5 archive (instead of default init values)
- Created make_fc_metadata_from_processing_config - there are still some things to iron out but its working on first synthetic test - Tuned some logic in tf_kernel
- add logic to get fc from h5 if exists - skip timeseries initialization in KernleDataset if fcs exist - skip timeseries decimation in tfk if fcs exist - had to swap between dataset and dataarray ion extract_band to use FC data (maybe due to h5 ref)
Previously was passing FCDecimationGroup but all checks are perfomed on the layer fc_group.metadata which is an object of type: mt_metadata.transfer_functions.processing.fourier_coefficients.decimation.Decimation So, update and rename the consistency check to: fcdecimation_has_fcs_for_processing. This is a staging commit, eventually this method will become a class method of mt_metadata.transfer_functions.processing.fourier_coefficients.decimation.Decimation
Fix issue 203a manual merge 3.8 Passing, others failing on doctest
Codecov Report
@@ Coverage Diff @@
## earthscope_tests #286 +/- ##
====================================================
- Coverage 77.31% 75.37% -1.95%
====================================================
Files 84 86 +2
Lines 5170 5534 +364
====================================================
+ Hits 3997 4171 +174
- Misses 1173 1363 +190
|
Use built-in method that i created a while ago. This commit has a bunch of commented lines (for posterity) at the top if process_mth5.py. These will be deleted and cleaned up now.
…args in mt_metadata
- replace boolean kwarg "remake_if_exists" with "force_make_mth5" - add test_create_then_use_stored_fcs_for_processing to test_fourier_coefficients (using a local import)
- latest fix is on fix_issue_157
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
FCs are now passing basic tests. Just wanted to leave a record of merging the updates into earthscope tests branch in case we need to go back. mth5_helpers.py is not merged (there are several functions in earthscope_tests not in FCs.