Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NOIRLab advanced search #1701

Open
wants to merge 14 commits into
base: main
Choose a base branch
from
Open

NOIRLab advanced search #1701

wants to merge 14 commits into from

Conversation

pothiers
Copy link
Contributor

Adds capability to search ANY metadata in Archived FITS file headers (in any HDU).

@pothiers
Copy link
Contributor Author

Milestone needs to be set on this (0.4.1). If its possible for me to do it, let me know how.

@pep8speaks
Copy link

pep8speaks commented Apr 16, 2020

Hello @pothiers! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2020-05-25 16:08:31 UTC

@codecov
Copy link

codecov bot commented Apr 16, 2020

Codecov Report

Merging #1701 into master will increase coverage by 0.29%.
The diff coverage is 28.30%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1701      +/-   ##
==========================================
+ Coverage   62.67%   62.97%   +0.29%     
==========================================
  Files         189      195       +6     
  Lines       15192    15295     +103     
==========================================
+ Hits         9522     9632     +110     
+ Misses       5670     5663       -7     
Impacted Files Coverage Δ
astroquery/noirlab/__init__.py 100.00% <ø> (ø)
astroquery/noirlab/core.py 40.00% <28.30%> (-18.98%) ⬇️
astroquery/alma/core.py 35.42% <0.00%> (-0.16%) ⬇️
astroquery/mast/__init__.py 100.00% <0.00%> (ø)
astroquery/mast/fpl.py
astroquery/mast/tesscut.py
astroquery/mast/utils.py 76.08% <0.00%> (ø)
astroquery/mast/services.py 78.03% <0.00%> (ø)
astroquery/mast/collections.py 92.10% <0.00%> (ø)
astroquery/mast/cutouts.py 78.76% <0.00%> (ø)
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 272bfb1...75e72a8. Read the comment docs.

@pothiers
Copy link
Contributor Author

I don't think the documentation is complete. I must have lost something in the shuffle of three PRs plus one previous version of this. Looking into it. (It should have lots of extra methods in the API)

Copy link
Member

@bsipocz bsipocz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The are a few comments, but nothing substantial with the exception of having asycn methods instead and rely on the decorator for the rest.

astroquery/noirlab/__init__.py Show resolved Hide resolved
astroquery/noirlab/core.py Outdated Show resolved Hide resolved
astroquery/noirlab/core.py Show resolved Hide resolved
astroquery/noirlab/tests/test_noirlab_remote.py Outdated Show resolved Hide resolved
@bsipocz
Copy link
Member

bsipocz commented Apr 23, 2020

@pothiers - It would be nice to have more substantial user facing documentation, with more examples. If you prefer it can be added in a follow-up PR once this is merged. E.g. the nice test coverage could be worked into a narrative docs, too.

Also, I see 3 failures that we won't see on CI as we don't run the remote tests for PRs to limit the stress on remote servers. For the record these are the failures I see.

Note for @keflavich and @ceb8 - we need to double check the test locally before merging this.

======================================================= FAILURES =======================================================
________________________________________ TestNoirlabClass.test_core_file_fields ________________________________________

self = <astroquery.noirlab.tests.test_noirlab_remote.TestNoirlabClass object at 0x135f57990>

    def test_core_file_fields(self):
        """List the available CORE FILE fields."""
        r = Noirlab().core_fields()
        actual = r
        print(f'DBG: test_core_file_fields={actual}')
        expected = exp.core_file_fields
>       assert actual == expected
E       AssertionError: assert [{'Field': 'a...loat64'}, ...] == {'archive_fil... type.'], ...}
E         Use -v to get the full diff

astroquery/noirlab/tests/test_noirlab_remote.py:93: AssertionError
------------------------------------------------- Captured stdout call -------------------------------------------------
DBG: test_core_file_fields=[{'Field': 'archive_filename', 'Type': 'str'}, {'Field': 'caldat', 'Type': 'datetime64'}, {'Field': 'date_obs_max', 'Type': 'datetime64'}, {'Field': 'date_obs_min', 'Type': 'datetime64'}, {'Field': 'dec_max', 'Type': 'np.float64'}, {'Field': 'dec_min', 'Type': 'np.float64'}, {'Field': 'depth', 'Type': 'np.float64'}, {'Field': 'exposure', 'Type': 'np.float64'}, {'Field': 'filesize', 'Type': 'np.int64'}, {'Field': 'ifilter', 'Type': 'category'}, {'Field': 'instrument', 'Type': 'category'}, {'Field': 'md5sum', 'Type': 'str'}, {'Field': 'obs_mode', 'Type': 'category'}, {'Field': 'obs_type', 'Type': 'category'}, {'Field': 'original_filename', 'Type': 'str'}, {'Field': 'proc_type', 'Type': 'category'}, {'Field': 'prod_type', 'Type': 'category'}, {'Field': 'proposal', 'Type': 'category'}, {'Field': 'ra_max', 'Type': 'np.float64'}, {'Field': 'ra_min', 'Type': 'np.float64'}, {'Field': 'release_date', 'Type': 'datetime64'}, {'Field': 'seeing', 'Type': 'np.float64'}, {'Field': 'site', 'Type': 'category'}, {'Field': 'survey', 'Type': 'category'}, {'Field': 'telescope', 'Type': 'category'}, {'Field': 'updated', 'Type': 'datetime64'}]
________________________________________ TestNoirlabClass.test_core_hdu_fields _________________________________________

self = <astroquery.noirlab.tests.test_noirlab_remote.TestNoirlabClass object at 0x135f1df10>

    def test_core_hdu_fields(self):
        """List the available CORE HDU fields."""
        r = Noirlab(which='hdu').core_fields()
        actual = r
        print(f'DBG: test_core_hdu_fields={actual}')
        expected = exp.core_hdu_fields
>       assert actual == expected
E       AssertionError: assert [{'Field': 'b...: 'str'}, ...] == {'boundary': ... match'], ...}
E         Use -v to get the full diff

astroquery/noirlab/tests/test_noirlab_remote.py:134: AssertionError
------------------------------------------------- Captured stdout call -------------------------------------------------
DBG: test_core_hdu_fields=[{'Field': 'boundary', 'Type': 'str'}, {'Field': 'dec', 'Type': 'np.float64'}, {'Field': 'dec_range', 'Type': 'str'}, {'Field': 'fitsfile', 'Type': 'str'}, {'Field': 'fitsfile__archive_filename', 'Type': 'str'}, {'Field': 'fitsfile__caldat', 'Type': 'str'}, {'Field': 'fitsfile__date_obs_max', 'Type': 'str'}, {'Field': 'fitsfile__date_obs_min', 'Type': 'str'}, {'Field': 'fitsfile__dec_max', 'Type': 'str'}, {'Field': 'fitsfile__dec_min', 'Type': 'str'}, {'Field': 'fitsfile__depth', 'Type': 'str'}, {'Field': 'fitsfile__exposure', 'Type': 'str'}, {'Field': 'fitsfile__filesize', 'Type': 'str'}, {'Field': 'fitsfile__ifilter', 'Type': 'str'}, {'Field': 'fitsfile__instrument', 'Type': 'str'}, {'Field': 'fitsfile__md5sum', 'Type': 'str'}, {'Field': 'fitsfile__obs_mode', 'Type': 'str'}, {'Field': 'fitsfile__obs_type', 'Type': 'str'}, {'Field': 'fitsfile__original_filename', 'Type': 'str'}, {'Field': 'fitsfile__proc_type', 'Type': 'str'}, {'Field': 'fitsfile__prod_type', 'Type': 'str'}, {'Field': 'fitsfile__proposal', 'Type': 'str'}, {'Field': 'fitsfile__ra_max', 'Type': 'str'}, {'Field': 'fitsfile__ra_min', 'Type': 'str'}, {'Field': 'fitsfile__release_date', 'Type': 'str'}, {'Field': 'fitsfile__seeing', 'Type': 'str'}, {'Field': 'fitsfile__site', 'Type': 'str'}, {'Field': 'fitsfile__survey', 'Type': 'str'}, {'Field': 'fitsfile__telescope', 'Type': 'str'}, {'Field': 'fitsfile__updated', 'Type': 'str'}, {'Field': 'hdu_idx', 'Type': 'np.int64'}, {'Field': 'id', 'Type': 'str'}, {'Field': 'ra', 'Type': 'np.float64'}, {'Field': 'ra_range', 'Type': 'str'}, {'Field': 'updated', 'Type': 'datetime64'}]
__________________________________________ TestNoirlabClass.test_categoricals __________________________________________

self = <astroquery.noirlab.tests.test_noirlab_remote.TestNoirlabClass object at 0x135f47750>

    def test_categoricals(self):
        """List categories."""
        r = Noirlab().categoricals()
        actual = r
        print(f'DBG: test_categoricals={actual}')
        expected = exp.categoricals
>       assert actual == expected
E       AssertionError: assert {'instruments...1', ...], ...} == {'collections...d', ...], ...}
E         Omitting 7 identical items, use -vv to show
E         Right contains 1 more item:
E         {'collections': []}
E         Use -v to get the full diff

astroquery/noirlab/tests/test_noirlab_remote.py:169: AssertionError
------------------------------------------------- Captured stdout call -------------------------------------------------
DBG: test_categoricals={'instruments': ['(p)odi', '90prime', 'andicam', 'arcoiris', 'arcon', 'bench', 'ccd_imager', 'chiron', 'cosmos', 'decam', 'echelle', 'falmingos', 'flamingos', 'goodman', 'goodman spectrograph', 'gtcam', 'hdi', 'ice', 'ispi', 'kosmos', 'minimo/ice', 'mop/ice', 'mosaic', 'mosaic3', 'mosaic_1', 'mosaic_1_1', 'mosaic_2', 'newfirm', 'osiris', 'sami', 'soi', 'spartan', 'spartan ir camera', 'triplespec', 'wfc', 'whirc', 'wildfire', 'y4kcam'], 'obsmodes': [], 'proctypes': ['instcal', 'mastercal', 'nota', 'projected', 'raw', 'resampled', 'skysub', 'stacked'], 'prodtypes': ['dqmask', 'expmap', 'graphics (size)', 'image', 'image 2nd version 1', 'image1', 'nota', 'resampled', 'weight', 'wtmap'], 'sites': ['cp', 'ct', 'kp', 'lp'], 'surveys': [], 'telescopes': ['bok23m', 'ct09m', 'ct13m', 'ct15m', 'ct1m', 'ct4m', 'ctlab', 'kp09m', 'kp21m', 'kp35m', 'kp4m', 'kpcf', 'lp25m', 'soar', 'wiyn']}
============================================ 3 failed, 12 passed in 17.38s =============================================

@pothiers
Copy link
Contributor Author

@pothiers - It would be nice to have more substantial user facing documentation, with more examples. If you prefer it can be added in a follow-up PR once this is merged. E.g. the nice test coverage could be worked into a narrative docs, too.

Have added more "advanced search" documentation.

@pothiers
Copy link
Contributor Author

@pothiers - It would be nice to have more substantial user facing documentation, with more examples. If you prefer it can be added in a follow-up PR once this is merged. E.g. the nice test coverage could be worked into a narrative docs, too.

Also, I see 3 failures that we won't see on CI as we don't run the remote tests for PRs to limit the stress on remote servers. For the record these are the failures I see.

Note for @keflavich and @ceb8 - we need to double check the test locally before merging this.

======================================================= FAILURES =======================================================
________________________________________ TestNoirlabClass.test_core_file_fields ________________________________________

self = <astroquery.noirlab.tests.test_noirlab_remote.TestNoirlabClass object at 0x135f57990>

    def test_core_file_fields(self):
        """List the available CORE FILE fields."""
        r = Noirlab().core_fields()
        actual = r
        print(f'DBG: test_core_file_fields={actual}')
        expected = exp.core_file_fields
>       assert actual == expected
E       AssertionError: assert [{'Field': 'a...loat64'}, ...] == {'archive_fil... type.'], ...}
E         Use -v to get the full diff

astroquery/noirlab/tests/test_noirlab_remote.py:93: AssertionError
------------------------------------------------- Captured stdout call -------------------------------------------------
DBG: test_core_file_fields=[{'Field': 'archive_filename', 'Type': 'str'}, {'Field': 'caldat', 'Type': 'datetime64'}, {'Field': 'date_obs_max', 'Type': 'datetime64'}, {'Field': 'date_obs_min', 'Type': 'datetime64'}, {'Field': 'dec_max', 'Type': 'np.float64'}, {'Field': 'dec_min', 'Type': 'np.float64'}, {'Field': 'depth', 'Type': 'np.float64'}, {'Field': 'exposure', 'Type': 'np.float64'}, {'Field': 'filesize', 'Type': 'np.int64'}, {'Field': 'ifilter', 'Type': 'category'}, {'Field': 'instrument', 'Type': 'category'}, {'Field': 'md5sum', 'Type': 'str'}, {'Field': 'obs_mode', 'Type': 'category'}, {'Field': 'obs_type', 'Type': 'category'}, {'Field': 'original_filename', 'Type': 'str'}, {'Field': 'proc_type', 'Type': 'category'}, {'Field': 'prod_type', 'Type': 'category'}, {'Field': 'proposal', 'Type': 'category'}, {'Field': 'ra_max', 'Type': 'np.float64'}, {'Field': 'ra_min', 'Type': 'np.float64'}, {'Field': 'release_date', 'Type': 'datetime64'}, {'Field': 'seeing', 'Type': 'np.float64'}, {'Field': 'site', 'Type': 'category'}, {'Field': 'survey', 'Type': 'category'}, {'Field': 'telescope', 'Type': 'category'}, {'Field': 'updated', 'Type': 'datetime64'}]
________________________________________ TestNoirlabClass.test_core_hdu_fields _________________________________________

self = <astroquery.noirlab.tests.test_noirlab_remote.TestNoirlabClass object at 0x135f1df10>

    def test_core_hdu_fields(self):
        """List the available CORE HDU fields."""
        r = Noirlab(which='hdu').core_fields()
        actual = r
        print(f'DBG: test_core_hdu_fields={actual}')
        expected = exp.core_hdu_fields
>       assert actual == expected
E       AssertionError: assert [{'Field': 'b...: 'str'}, ...] == {'boundary': ... match'], ...}
E         Use -v to get the full diff

astroquery/noirlab/tests/test_noirlab_remote.py:134: AssertionError
------------------------------------------------- Captured stdout call -------------------------------------------------
DBG: test_core_hdu_fields=[{'Field': 'boundary', 'Type': 'str'}, {'Field': 'dec', 'Type': 'np.float64'}, {'Field': 'dec_range', 'Type': 'str'}, {'Field': 'fitsfile', 'Type': 'str'}, {'Field': 'fitsfile__archive_filename', 'Type': 'str'}, {'Field': 'fitsfile__caldat', 'Type': 'str'}, {'Field': 'fitsfile__date_obs_max', 'Type': 'str'}, {'Field': 'fitsfile__date_obs_min', 'Type': 'str'}, {'Field': 'fitsfile__dec_max', 'Type': 'str'}, {'Field': 'fitsfile__dec_min', 'Type': 'str'}, {'Field': 'fitsfile__depth', 'Type': 'str'}, {'Field': 'fitsfile__exposure', 'Type': 'str'}, {'Field': 'fitsfile__filesize', 'Type': 'str'}, {'Field': 'fitsfile__ifilter', 'Type': 'str'}, {'Field': 'fitsfile__instrument', 'Type': 'str'}, {'Field': 'fitsfile__md5sum', 'Type': 'str'}, {'Field': 'fitsfile__obs_mode', 'Type': 'str'}, {'Field': 'fitsfile__obs_type', 'Type': 'str'}, {'Field': 'fitsfile__original_filename', 'Type': 'str'}, {'Field': 'fitsfile__proc_type', 'Type': 'str'}, {'Field': 'fitsfile__prod_type', 'Type': 'str'}, {'Field': 'fitsfile__proposal', 'Type': 'str'}, {'Field': 'fitsfile__ra_max', 'Type': 'str'}, {'Field': 'fitsfile__ra_min', 'Type': 'str'}, {'Field': 'fitsfile__release_date', 'Type': 'str'}, {'Field': 'fitsfile__seeing', 'Type': 'str'}, {'Field': 'fitsfile__site', 'Type': 'str'}, {'Field': 'fitsfile__survey', 'Type': 'str'}, {'Field': 'fitsfile__telescope', 'Type': 'str'}, {'Field': 'fitsfile__updated', 'Type': 'str'}, {'Field': 'hdu_idx', 'Type': 'np.int64'}, {'Field': 'id', 'Type': 'str'}, {'Field': 'ra', 'Type': 'np.float64'}, {'Field': 'ra_range', 'Type': 'str'}, {'Field': 'updated', 'Type': 'datetime64'}]
__________________________________________ TestNoirlabClass.test_categoricals __________________________________________

self = <astroquery.noirlab.tests.test_noirlab_remote.TestNoirlabClass object at 0x135f47750>

    def test_categoricals(self):
        """List categories."""
        r = Noirlab().categoricals()
        actual = r
        print(f'DBG: test_categoricals={actual}')
        expected = exp.categoricals
>       assert actual == expected
E       AssertionError: assert {'instruments...1', ...], ...} == {'collections...d', ...], ...}
E         Omitting 7 identical items, use -vv to show
E         Right contains 1 more item:
E         {'collections': []}
E         Use -v to get the full diff

astroquery/noirlab/tests/test_noirlab_remote.py:169: AssertionError
------------------------------------------------- Captured stdout call -------------------------------------------------
DBG: test_categoricals={'instruments': ['(p)odi', '90prime', 'andicam', 'arcoiris', 'arcon', 'bench', 'ccd_imager', 'chiron', 'cosmos', 'decam', 'echelle', 'falmingos', 'flamingos', 'goodman', 'goodman spectrograph', 'gtcam', 'hdi', 'ice', 'ispi', 'kosmos', 'minimo/ice', 'mop/ice', 'mosaic', 'mosaic3', 'mosaic_1', 'mosaic_1_1', 'mosaic_2', 'newfirm', 'osiris', 'sami', 'soi', 'spartan', 'spartan ir camera', 'triplespec', 'wfc', 'whirc', 'wildfire', 'y4kcam'], 'obsmodes': [], 'proctypes': ['instcal', 'mastercal', 'nota', 'projected', 'raw', 'resampled', 'skysub', 'stacked'], 'prodtypes': ['dqmask', 'expmap', 'graphics (size)', 'image', 'image 2nd version 1', 'image1', 'nota', 'resampled', 'weight', 'wtmap'], 'sites': ['cp', 'ct', 'kp', 'lp'], 'surveys': [], 'telescopes': ['bok23m', 'ct09m', 'ct13m', 'ct15m', 'ct1m', 'ct4m', 'ctlab', 'kp09m', 'kp21m', 'kp35m', 'kp4m', 'kpcf', 'lp25m', 'soar', 'wiyn']}
============================================ 3 failed, 12 passed in 17.38s =============================================

When I did:

cd ~/astroquery/astroquery/noirlab
pytest --remote-data 

I got no errors. But did get them when doing:

cd ~/astroquery
python setup.py test -P noirlab -R

I thought the two would be equivalent for module only checks. I'll change my checklist to only use the latter test invocation.

@pothiers
Copy link
Contributor Author

Because there where changes to my PR1701 I merged it locally. I now think I should have rebased. I haven't yet pushed my updates (including conflict resolution). Do I need to somehow untangle to avoid the extra commit logs that will otherwise show? If so, I'll likely need help with the untangling.

@pothiers
Copy link
Contributor Author

@pothiers - It would be nice to have more substantial user facing documentation, with more examples. If you prefer it can be added in a follow-up PR once this is merged. E.g. the nice test coverage could be worked into a narrative docs, too.

Also, I see 3 failures that we won't see on CI as we don't run the remote tests for PRs to limit the stress on remote servers. For the record these are the failures I see.

Note for @keflavich and @ceb8 - we need to double check the test locally before merging this.

======================================================= FAILURES =======================================================
________________________________________ TestNoirlabClass.test_core_file_fields ________________________________________

self = <astroquery.noirlab.tests.test_noirlab_remote.TestNoirlabClass object at 0x135f57990>

    def test_core_file_fields(self):
        """List the available CORE FILE fields."""
        r = Noirlab().core_fields()
        actual = r
        print(f'DBG: test_core_file_fields={actual}')
        expected = exp.core_file_fields
>       assert actual == expected
E       AssertionError: assert [{'Field': 'a...loat64'}, ...] == {'archive_fil... type.'], ...}
E         Use -v to get the full diff

astroquery/noirlab/tests/test_noirlab_remote.py:93: AssertionError
------------------------------------------------- Captured stdout call -------------------------------------------------
DBG: test_core_file_fields=[{'Field': 'archive_filename', 'Type': 'str'}, {'Field': 'caldat', 'Type': 'datetime64'}, {'Field': 'date_obs_max', 'Type': 'datetime64'}, {'Field': 'date_obs_min', 'Type': 'datetime64'}, {'Field': 'dec_max', 'Type': 'np.float64'}, {'Field': 'dec_min', 'Type': 'np.float64'}, {'Field': 'depth', 'Type': 'np.float64'}, {'Field': 'exposure', 'Type': 'np.float64'}, {'Field': 'filesize', 'Type': 'np.int64'}, {'Field': 'ifilter', 'Type': 'category'}, {'Field': 'instrument', 'Type': 'category'}, {'Field': 'md5sum', 'Type': 'str'}, {'Field': 'obs_mode', 'Type': 'category'}, {'Field': 'obs_type', 'Type': 'category'}, {'Field': 'original_filename', 'Type': 'str'}, {'Field': 'proc_type', 'Type': 'category'}, {'Field': 'prod_type', 'Type': 'category'}, {'Field': 'proposal', 'Type': 'category'}, {'Field': 'ra_max', 'Type': 'np.float64'}, {'Field': 'ra_min', 'Type': 'np.float64'}, {'Field': 'release_date', 'Type': 'datetime64'}, {'Field': 'seeing', 'Type': 'np.float64'}, {'Field': 'site', 'Type': 'category'}, {'Field': 'survey', 'Type': 'category'}, {'Field': 'telescope', 'Type': 'category'}, {'Field': 'updated', 'Type': 'datetime64'}]
________________________________________ TestNoirlabClass.test_core_hdu_fields _________________________________________

self = <astroquery.noirlab.tests.test_noirlab_remote.TestNoirlabClass object at 0x135f1df10>

    def test_core_hdu_fields(self):
        """List the available CORE HDU fields."""
        r = Noirlab(which='hdu').core_fields()
        actual = r
        print(f'DBG: test_core_hdu_fields={actual}')
        expected = exp.core_hdu_fields
>       assert actual == expected
E       AssertionError: assert [{'Field': 'b...: 'str'}, ...] == {'boundary': ... match'], ...}
E         Use -v to get the full diff

astroquery/noirlab/tests/test_noirlab_remote.py:134: AssertionError
------------------------------------------------- Captured stdout call -------------------------------------------------
DBG: test_core_hdu_fields=[{'Field': 'boundary', 'Type': 'str'}, {'Field': 'dec', 'Type': 'np.float64'}, {'Field': 'dec_range', 'Type': 'str'}, {'Field': 'fitsfile', 'Type': 'str'}, {'Field': 'fitsfile__archive_filename', 'Type': 'str'}, {'Field': 'fitsfile__caldat', 'Type': 'str'}, {'Field': 'fitsfile__date_obs_max', 'Type': 'str'}, {'Field': 'fitsfile__date_obs_min', 'Type': 'str'}, {'Field': 'fitsfile__dec_max', 'Type': 'str'}, {'Field': 'fitsfile__dec_min', 'Type': 'str'}, {'Field': 'fitsfile__depth', 'Type': 'str'}, {'Field': 'fitsfile__exposure', 'Type': 'str'}, {'Field': 'fitsfile__filesize', 'Type': 'str'}, {'Field': 'fitsfile__ifilter', 'Type': 'str'}, {'Field': 'fitsfile__instrument', 'Type': 'str'}, {'Field': 'fitsfile__md5sum', 'Type': 'str'}, {'Field': 'fitsfile__obs_mode', 'Type': 'str'}, {'Field': 'fitsfile__obs_type', 'Type': 'str'}, {'Field': 'fitsfile__original_filename', 'Type': 'str'}, {'Field': 'fitsfile__proc_type', 'Type': 'str'}, {'Field': 'fitsfile__prod_type', 'Type': 'str'}, {'Field': 'fitsfile__proposal', 'Type': 'str'}, {'Field': 'fitsfile__ra_max', 'Type': 'str'}, {'Field': 'fitsfile__ra_min', 'Type': 'str'}, {'Field': 'fitsfile__release_date', 'Type': 'str'}, {'Field': 'fitsfile__seeing', 'Type': 'str'}, {'Field': 'fitsfile__site', 'Type': 'str'}, {'Field': 'fitsfile__survey', 'Type': 'str'}, {'Field': 'fitsfile__telescope', 'Type': 'str'}, {'Field': 'fitsfile__updated', 'Type': 'str'}, {'Field': 'hdu_idx', 'Type': 'np.int64'}, {'Field': 'id', 'Type': 'str'}, {'Field': 'ra', 'Type': 'np.float64'}, {'Field': 'ra_range', 'Type': 'str'}, {'Field': 'updated', 'Type': 'datetime64'}]
__________________________________________ TestNoirlabClass.test_categoricals __________________________________________

self = <astroquery.noirlab.tests.test_noirlab_remote.TestNoirlabClass object at 0x135f47750>

    def test_categoricals(self):
        """List categories."""
        r = Noirlab().categoricals()
        actual = r
        print(f'DBG: test_categoricals={actual}')
        expected = exp.categoricals
>       assert actual == expected
E       AssertionError: assert {'instruments...1', ...], ...} == {'collections...d', ...], ...}
E         Omitting 7 identical items, use -vv to show
E         Right contains 1 more item:
E         {'collections': []}
E         Use -v to get the full diff

astroquery/noirlab/tests/test_noirlab_remote.py:169: AssertionError
------------------------------------------------- Captured stdout call -------------------------------------------------
DBG: test_categoricals={'instruments': ['(p)odi', '90prime', 'andicam', 'arcoiris', 'arcon', 'bench', 'ccd_imager', 'chiron', 'cosmos', 'decam', 'echelle', 'falmingos', 'flamingos', 'goodman', 'goodman spectrograph', 'gtcam', 'hdi', 'ice', 'ispi', 'kosmos', 'minimo/ice', 'mop/ice', 'mosaic', 'mosaic3', 'mosaic_1', 'mosaic_1_1', 'mosaic_2', 'newfirm', 'osiris', 'sami', 'soi', 'spartan', 'spartan ir camera', 'triplespec', 'wfc', 'whirc', 'wildfire', 'y4kcam'], 'obsmodes': [], 'proctypes': ['instcal', 'mastercal', 'nota', 'projected', 'raw', 'resampled', 'skysub', 'stacked'], 'prodtypes': ['dqmask', 'expmap', 'graphics (size)', 'image', 'image 2nd version 1', 'image1', 'nota', 'resampled', 'weight', 'wtmap'], 'sites': ['cp', 'ct', 'kp', 'lp'], 'surveys': [], 'telescopes': ['bok23m', 'ct09m', 'ct13m', 'ct15m', 'ct1m', 'ct4m', 'ctlab', 'kp09m', 'kp21m', 'kp35m', 'kp4m', 'kpcf', 'lp25m', 'soar', 'wiyn']}
============================================ 3 failed, 12 passed in 17.38s =============================================

Failed tests were due to new release of our server that happen to come after I submitted working tests. Fixed locally but awaiting guidance on how to best combine changes to PR

@keflavich
Copy link
Contributor

@pothiers yes, let's rebase this and try to disentangle the histories. Ping me on slack if you need help

@bsipocz
Copy link
Member

bsipocz commented Apr 29, 2020

@keflavich - git history-wise this is all right now

docs/noirlab/noirlab.rst Outdated Show resolved Hide resolved
@bsipocz
Copy link
Member

bsipocz commented Apr 29, 2020

CI has 2 test failing with astropy-dev due to this: astropy/astropy#10227, thus it shouldn't be a blocker for this PR. However the async/sync issue should still be addressed before merging.

@bsipocz
Copy link
Member

bsipocz commented Apr 29, 2020

Have added more "advanced search" documentation.

Thank you. What I had in mind is to have a bit more exact examples, driven by simple science cases to showcase the functionality of the module (not necessarily just the advanced search), otherwise I'm afraid the users won't know what the module is capable of.
This can be addressed in a follow-up later.

@ceb8
Copy link
Member

ceb8 commented Apr 29, 2020

I'm getting two failures when I run the remote tests, both the same:

AttributeError: 'Table' object has no attribute 'pformat_all'

@pothiers pothiers requested a review from keflavich May 13, 2020 17:32
Comment on lines 182 to 187
def aux_fields(self, instrument, proctype, cache=True):
"""List the available AUX fields. AUX fields are ANY fields in the
Archive FITS files that are not core DB fields. These are generally
common to a single Instrument, Proctype combination. AUX fields are
slower to search than CORE fields. """
url = f'{self._adsa_url}/{instrument}/{proctype}/'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since instrument and proctype are user-entered parameters, please document them here (and maybe consider adding some sort of validity check)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All of these functions are supported by web-services which do the validation. We have people accessing directly via the web-services. Doing validation here would be redundant (and require DB access since what is valid depends on DB. I've added note in doc that says the acceptable values come from CATEGORICALS method.

Copy link
Contributor

@keflavich keflavich May 18, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need the parameters documented here, in the docstring.

If a function takes parameters, as this one does, those parameters need to be documented in the docstring, i.e.:

Parameters
----------
instrument : str
    Instrument name, see <categoricals?> for details

etc.

return response.json()

@class_or_instance
def query_metadata(self, qspec, limit=1000, cache=True):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this could also be async'd, but nbd if it isn't

it does, however, need some docs!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean, this function requires a docstring.

astroquery/noirlab/core.py Show resolved Hide resolved
response.raise_for_status()
return astropy.table.Table(rows=response.json())

def retrieve(self, fileid, cache=True):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need docstring

astroquery/noirlab/core.py Show resolved Hide resolved
astroquery/noirlab/core.py Outdated Show resolved Hide resolved
@keflavich
Copy link
Contributor

The idea behind async_to_sync is to avoid duplicate code. In brief:

  • an _async method should return a requests response object
  • a _parse_result function should be defined that takes a response object and parses it into an astropy table
  • if async_to_sync is used, you should not need to define query_<blah>, you should only define query_<blah>_async.

In the current version of this code, async_to_sync has no effect because query_region_async and query_region are both defined explicitly. You don't need to use async_to_sync - it is, admittedly, a bit confusing. However, there is no reason to duplicate code in query_region and in query_region_async. I'll make an explicit code suggestion to illustrate this.

Comment on lines 103 to 109
self._validate_version()
ra, dec = coordinate.to_string('decimal').split()
url = f'{self.url}?POS={ra},{dec}&SIZE={radius}&format=json'
url = f'{self.siaurl}?POS={ra},{dec}&SIZE={radius}&format=json'
response = self._request('GET', url,
timeout=self.TIMEOUT,
cache=cache)
response.raise_for_status()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest replacing all of this with:

response = self.query_region_async(coordinate, radius=radius, cache=cache)

@bsipocz bsipocz modified the milestones: v0.4.1, v0.4.2 Jun 18, 2020
@bsipocz bsipocz modified the milestones: v0.4.2, v0.4.3 May 15, 2021
@bsipocz bsipocz modified the milestones: v0.4.3, v0.4.4 Jul 7, 2021
@bsipocz bsipocz removed this from the v0.4.4 milestone Nov 13, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants