Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Nightlights data #535

Open
olafveerman opened this issue Dec 22, 2020 · 4 comments
Open

Update Nightlights data #535

olafveerman opened this issue Dec 22, 2020 · 4 comments
Labels
data Data processing
Milestone

Comments

@olafveerman
Copy link
Contributor

New Nightlight data through Dec 1 was added to the S3 bucket s3://covid-eo-blackmarble/

  • COVID_Dashboard_BMHD_Jan_Nov_2020.tar.gz
  • COVID_Dashboard_VNP46A2_Jan_Nov_2020.tar.gz
@olafveerman olafveerman added the data Data processing label Dec 22, 2020
@olafveerman olafveerman added this to the v1.10.0 milestone Dec 22, 2020
@leothomas
Copy link
Collaborator

@olafveerman update on re-processing of the nightlight's dataset:

  • BMHD (30m monthly dataset) is updated and in a staging folder in S3 (s3://covid-eo-data/bmhd_30m_monthly_staging)
  • VNP46A2 (500m daily dataset) is almost entirely updated in a staging folder in S3 (s3://covid-eo-data/bm_500m_daily_staging). There were some image archives (.img.tar.gz) that I was unable to extract (see below). It looks like those files are each only 45 bytes. I would love a second pair of eyes on this see if there was a processing error somewhere along the chain.
Size, Last Updated, Filepath (relative)
45B, Dec  4 01:07, ./h06v05/2020/012/near_nadir/VNP46A2_V011.h06v05.A2020012.5000.22.22_68.73_V37.img.tar.gz
45B, Dec  5 03:00, ./h18v03/2020/007/near_nadir/VNP46A2_V011.h18v03.A2020007.5000.36.85_51.77_V37.img.tar.gz
45B, Dec  5 03:01, ./h18v03/2020/009/near_nadir/VNP46A2_V011.h18v03.A2020009.5000.20.21_30.76_V37.img.tar.gz
45B, Dec  5 03:01, ./h18v03/2020/008/near_nadir/VNP46A2_V011.h18v03.A2020008.5000.23.84_58.21_V37.img.tar.gz
45B, Dec  5 02:59, ./h18v03/2020/006/near_nadir/VNP46A2_V011.h18v03.A2020006.5000.48.58_45.54_V37.img.tar.gz
45B, Dec  5 03:03, ./h18v03/2020/011/near_nadir/VNP46A2_V011.h18v03.A2020011.5000.15.44_52.94_V37.img.tar.gz
45B, Dec  5 03:02, ./h18v03/2020/010/near_nadir/VNP46A2_V011.h18v03.A2020010.5000.2.25_31.58_V37.img.tar.gz
45B, Dec  4 03:52, ./h29v04/2020/011/near_nadir/VNP46A2_V011.h29v04.A2020011.5000.13.60_69.20_V37.img.tar.gz
45B, Dec  4 03:52, ./h29v04/2020/011/near_nadir/VNP46A2_V011.h29v04.A2020011.5000.13.60_69.20_V37.img.tar.gz
45B, Dec  4 01:09, ./h10v04/2020/012/near_nadir/VNP46A2_V011.h10v04.A2020012.5000.20.29_67.99_V37.img.tar.gz
45B, Dec  4 01:09, ./h10v04/2020/011/near_nadir/VNP46A2_V011.h10v04.A2020011.5000.6.92_65.39_V37.img.tar.gz
45B, Dec  4 00:28, ./h05v05/2020/012/near_nadir/VNP46A2_V011.h05v05.A2020012.5000.22.22_68.73_V37.img.tar.gz
45B, Dec  4 01:07, ./h06v05/2020/012/near_nadir/VNP46A2_V011.h06v05.A2020012.5000.22.22_68.73_V37.img.tar.gz
45B, Dec  5 03:00, ./h18v03/2020/007/near_nadir/VNP46A2_V011.h18v03.A2020007.5000.36.85_51.77_V37.img.tar.gz
45B, Dec  5 03:01, ./h18v03/2020/009/near_nadir/VNP46A2_V011.h18v03.A2020009.5000.20.21_30.76_V37.img.tar.gz
45B, Dec  5 03:01, ./h18v03/2020/008/near_nadir/VNP46A2_V011.h18v03.A2020008.5000.23.84_58.21_V37.img.tar.gz
45B, Dec  5 02:59, ./h18v03/2020/006/near_nadir/VNP46A2_V011.h18v03.A2020006.5000.48.58_45.54_V37.img.tar.gz
45B, Dec  5 03:03, ./h18v03/2020/011/near_nadir/VNP46A2_V011.h18v03.A2020011.5000.15.44_52.94_V37.img.tar.gz
45B, Dec  5 03:02, ./h18v03/2020/010/near_nadir/VNP46A2_V011.h18v03.A2020010.5000.2.25_31.58_V37.img.tar.gz
45B, Dec  4 03:52, ./h29v04/2020/011/near_nadir/VNP46A2_V011.h29v04.A2020011.5000.13.60_69.20_V37.img.tar.gz
45B, Dec  4 03:52, ./h29v04/2020/011/near_nadir/VNP46A2_V011.h29v04.A2020011.5000.13.60_69.20_V37.img.tar.gz
45B, Dec  4 01:09, ./h10v04/2020/012/near_nadir/VNP46A2_V011.h10v04.A2020012.5000.20.29_67.99_V37.img.tar.gz
45B, Dec  4 01:09, ./h10v04/2020/011/near_nadir/VNP46A2_V011.h10v04.A2020011.5000.6.92_65.39_V37.img.tar.gz
45B, Dec  4 00:28, ./h05v05/2020/012/near_nadir/VNP46A2_V011.h05v05.A2020012.5000.22.22_68.73_V37.img.tar.gz
45B, Dec  4 01:07, ./h06v05/2020/012/near_nadir/VNP46A2_V011.h06v05.A2020012.5000.22.22_68.73_V37.img.tar.gz
45B, Dec  5 03:00, ./h18v03/2020/007/near_nadir/VNP46A2_V011.h18v03.A2020007.5000.36.85_51.77_V37.img.tar.gz
45B, Dec  5 03:01, ./h18v03/2020/009/near_nadir/VNP46A2_V011.h18v03.A2020009.5000.20.21_30.76_V37.img.tar.gz
45B, Dec  5 03:01, ./h18v03/2020/008/near_nadir/VNP46A2_V011.h18v03.A2020008.5000.23.84_58.21_V37.img.tar.gz
45B, Dec  5 02:59, ./h18v03/2020/006/near_nadir/VNP46A2_V011.h18v03.A2020006.5000.48.58_45.54_V37.img.tar.gz
45B, Dec  5 03:03, ./h18v03/2020/011/near_nadir/VNP46A2_V011.h18v03.A2020011.5000.15.44_52.94_V37.img.tar.gz
45B, Dec  5 03:02, ./h18v03/2020/010/near_nadir/VNP46A2_V011.h18v03.A2020010.5000.2.25_31.58_V37.img.tar.gz
45B, Dec  4 03:52, ./h29v04/2020/011/near_nadir/VNP46A2_V011.h29v04.A2020011.5000.13.60_69.20_V37.img.tar.gz
45B, Dec  4 03:52, ./h29v04/2020/011/near_nadir/VNP46A2_V011.h29v04.A2020011.5000.13.60_69.20_V37.img.tar.gz
45B, Dec  4 01:09, ./h10v04/2020/012/near_nadir/VNP46A2_V011.h10v04.A2020012.5000.20.29_67.99_V37.img.tar.gz
45B, Dec  4 01:09, ./h10v04/2020/011/near_nadir/VNP46A2_V011.h10v04.A2020011.5000.6.92_65.39_V37.img.tar.gz
45B, Dec  4 00:28, ./h05v05/2020/012/near_nadir/VNP46A2_V011.h05v05.A2020012.5000.22.22_68.73_V37.img.tar.gz

@olafveerman
Copy link
Contributor Author

Nightlight data is updated.
Since the missing data is not breaking the frontend and it will be added in a future release, we can close this issue.

@olafveerman
Copy link
Contributor Author

Issue with missing data should be solved in the latest data drop.

@leothomas leothomas reopened this Feb 16, 2021
@leothomas
Copy link
Collaborator

Re-opening this issue:

Last data drop included tiles: h30v05 , h31v05 and h32v05 (all three tiles were associated with the Tokyo spotlight but cover the area between Tokyo and Seoul, with the tileh30v05 being over Seoul)

The updated data drop doesn't contain the h30v05 tile anymore. I can process without this tile with no problem, but I wanted to flag this in case it's a technical issue to solve.

Also, there are no longer any header (.hdr) files with each tile's daily tiff. Again I can process without the header, but I wanted to raise it in case it's an issue.

The processed data (without header files and h30v05 tiles) can be found at s3://covid-eo-data/bm_500m_daily_staging

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
data Data processing
Projects
None yet
Development

No branches or pull requests

2 participants