Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MUR SST Kerchunk Recipe #259

Draft
wants to merge 38 commits into
base: master
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
8e77f78
MUR SST Kerchunk Recipe
sbquinlan Oct 26, 2023
37000c6
Use filepattern for URLs and replace custom methods with WriteCombine…
abarciauskas-bgse Jan 12, 2024
1f0218b
Fix linting
abarciauskas-bgse Jan 12, 2024
fe4f047
Make protocol an environment variable
abarciauskas-bgse Jan 16, 2024
ca7c666
Use protocol envionement variable
abarciauskas-bgse Jan 16, 2024
7897490
Linting
abarciauskas-bgse Jan 17, 2024
b82e52b
Update recipe.py
abarciauskas-bgse Jan 24, 2024
e90707b
Update recipe.py
abarciauskas-bgse Jan 25, 2024
32b7c48
Update recipe.py
abarciauskas-bgse Jan 25, 2024
40e2885
Update recipe.py
abarciauskas-bgse Jan 25, 2024
26af6c7
Update recipe.py
abarciauskas-bgse Jan 25, 2024
df6ec0e
Update recipe.py
abarciauskas-bgse Jan 25, 2024
00f5992
Update recipe.py
abarciauskas-bgse Jan 25, 2024
31d6b9e
Update recipe.py
abarciauskas-bgse Jan 25, 2024
01f9ced
Update transforms
abarciauskas-bgse Jan 25, 2024
ece4650
Comment out failing transform
abarciauskas-bgse Jan 26, 2024
b4e2fbb
Add missing strings
abarciauskas-bgse Jan 29, 2024
02d80c2
Update requirements.txt
abarciauskas-bgse Jan 29, 2024
d270f9b
Remove consolidate metadata
abarciauskas-bgse Jan 29, 2024
b56af98
Update recipe.py
abarciauskas-bgse Feb 2, 2024
293c453
Update recipe.py
abarciauskas-bgse Feb 6, 2024
78bb384
comment out validate
abarciauskas-bgse Feb 6, 2024
7204f67
Update recipe.py
abarciauskas-bgse Feb 6, 2024
30cad0a
Update recipe.py
abarciauskas-bgse Feb 6, 2024
bc075dc
Update recipe.py
abarciauskas-bgse Feb 6, 2024
88394b4
Update recipe.py
abarciauskas-bgse Feb 6, 2024
f881b30
Update recipe.py
abarciauskas-bgse Feb 6, 2024
bd4a3d6
Update recipe.py
abarciauskas-bgse Feb 7, 2024
b97b227
Update recipe.py
abarciauskas-bgse Feb 7, 2024
271d320
Update recipe.py
abarciauskas-bgse Feb 7, 2024
2c411b0
Add requirements, recipe with scripts for running w/o runner
abarciauskas-bgse Feb 28, 2024
f62d830
Fix merge conflicts
abarciauskas-bgse Feb 28, 2024
36f2e71
Remove code for running without runner
abarciauskas-bgse Feb 28, 2024
bc3f2df
Fix fsspec_open_kwargs
abarciauskas-bgse Feb 28, 2024
b5e1b18
remove validation
Feb 28, 2024
46faa71
Merge pull request #4 from developmentseed/mursst-kerchunk-gcorradini
abarciauskas-bgse Feb 28, 2024
a4da5b6
Update recipe.py
abarciauskas-bgse Feb 28, 2024
644c753
Update recipe.py
abarciauskas-bgse Feb 28, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update recipe.py
abarciauskas-bgse authored Feb 6, 2024
commit f881b303804c3043ec1b6798a213ab4f613972fd
8 changes: 5 additions & 3 deletions recipes/mursst/recipe.py
Original file line number Diff line number Diff line change
@@ -162,8 +162,9 @@ def expand(
) -> beam.PCollection:
return pcoll | beam.Map(self._validate, expected_dims=self.expected_dims)

auth_args = earthdata_auth(ED_USERNAME, ED_PASSWORD)

auth_args = earthdata_auth(ED_USERNAME, ED_PASSWORD)

recipe = (
beam.Create(pattern.items())
| OpenWithKerchunk(
@@ -173,12 +174,13 @@ def expand(
inline_threshold=6000,
storage_options=auth_args,
)
| FilterVars(keep={*pattern.concat_dims, *IDENTICAL_DIMS, *SELECTED_VARS})
| WriteCombinedReference(

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This depends on the main branch of pangeo-forge-recipes

concat_dims=CONCAT_DIMS,
identical_dims=IDENTICAL_DIMS,
store_name=SHORT_NAME,
# for running without a runner, use this target_root
# target_root=fs_target,
mzz_kwargs={'coo_map': {"time": "cf:time"}, 'inline_threshold': 0}
)
#| ValidateDatasetDimensions(expected_dims={'time': None, 'lat': (-90, 90), 'lon': (-180, 180)})
| ValidateDatasetDimensions(expected_dims={'time': None, 'lat': (-90, 90), 'lon': (-180, 180)})
)