Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Global cropmap production V2 - Warning : less than 3 off-swath acquisitions #79

Open
romain-bdc opened this issue Dec 1, 2022 · 1 comment
Open

Global cropmap production V2 - Warning : less than 3 off-swath acquisitions #79

romain-bdc opened this issue Dec 1, 2022 · 1 comment

Comments

@romain-bdc
Copy link

For example for the tile 04QDK_004 :

ewoc_classif 04QDK c728b264-5c97-4f4c-81fe-1500d4c4dfbd_20088_20220925060043 --end-season-year 2021 --cropland-model-version v700 --croptype-model-version v720 --irr-model-version v420 --block-ids 4 --ewoc-detector cropland --upload-block False

we have following warning :

Start of processing
2022-12-01 14:22:54.702 | INFO     | 139633939470144:worldcereal.utils.aez:get_matching_aez_id:92 - Matched AEZ: zoneID = 20088 | groupID = 20000
[2022-12-01 14:22:54] WARNING:fiona._env:File /vsimem/61355faf4a3e4e04bc600d5162300c31 has GPKG application_id, but non conformant file extension
2022-12-01 14:22:58.025 | INFO     | 139633939470144:worldcereal.utils:get_matching_realm_id:290 - Matched REALM: 8
2022-12-01 14:22:58.168 | INFO     | 139633939470144:worldcereal.seasons:_get_cropland_processing_dates:237 - Derived start_date: 2020-09-17
2022-12-01 14:22:58.168 | INFO     | 139633939470144:worldcereal.seasons:_get_cropland_processing_dates:238 - Derived end_date: 2021-09-17
2022-12-01 14:22:58.169 | INFO     | 139633939470144:worldcereal.worldcereal_products:_get_block_size:62 - Found `EWOC_BLOCKSIZE` environment variable: 1024
2022-12-01 14:22:58.230 | INFO     | 139633939470144:worldcereal.worldcereal_products:run_tile:445 - Getting processing tuples ...
2022-12-01 14:22:58.231 | INFO     | 139633939470144:worldcereal.worldcereal_products:run_tile:476 - Running in serial ...
2022-12-01 14:22:58.236 | INFO     | 139633939470144:satio.utils.logs:wrapper_exitlog:73 - Starting processing of 04QDK_004.
2022-12-01 14:22:58.236 | INFO     | 139633939470144:worldcereal.worldcereal_products:_run_block:239 - --------------------------------------------------
2022-12-01 14:22:58.236 | INFO     | 139633939470144:worldcereal.worldcereal_products:_run_block:240 - Starting processing block: 04QDK_004
2022-12-01 14:22:58.236 | INFO     | 139633939470144:worldcereal.worldcereal_products:_run_block:241 - --------------------------------------------------
2022-12-01 14:22:58.237 | INFO     | 139633939470144:worldcereal.worldcereal_products:_run_block:242 - PARAMETERS:
2022-12-01 14:22:58.237 | INFO     | 139633939470144:worldcereal.worldcereal_products:_run_block:244 - year: 2021
2022-12-01 14:22:58.237 | INFO     | 139633939470144:worldcereal.worldcereal_products:_run_block:244 - season: annual
2022-12-01 14:22:58.237 | INFO     | 139633939470144:worldcereal.worldcereal_products:_run_block:244 - featuresettings: {'settings': {'OPTICAL': {'bands': ['B02', 'B03', 'B04', 'B05', 'B07', 'B08', 'B11', 'B12'], 'rsis': ['ndvi', 'anir', 'ndmi', 'ndwi', 'ndre1', 'ndre5', 'ndgi'], 'composite': {'freq': 10, 'window': 20, 'start': None, 'end': None}, 'smooth_ndvi': False, 'mask': {'erode_r': 3, 'dilate_r': 13, 'mask_values': [0, 1, 3, 8, 9, 10, 11], 'multitemporal': False}}, 'SAR': {'bands': ['VV', 'VH'], 'rsis': ['rvi'], 'composite': {'freq': 12, 'window': 18, 'mode': 'mean', 'start': None, 'end': None}}}, 'features_meta': {'OPTICAL': {'percentile_iqr': {'function': <function percentile_iqr at 0x7efed6d0d670>, 'parameters': {'q': [10, 50, 90], 'iqr': [25, 75]}, 'names': ['p10', 'p50', 'p90', 'iqr']}, 'tsteps': {'function': <function tsteps at 0x7efe9a982a60>, 'parameters': {'n_steps': 6}, 'names': ['ts0', 'ts1', 'ts2', 'ts3', 'ts4', 'ts5'], 'bands': ['ndvi']}, 'std': {'function': <function std at 0x7efed6d0d700>, 'names': ['std']}, 'skew': {'function': <function skewness at 0x7efed6d0d820>, 'names': ['skew']}, 'sen2agri_temp_feat': {'function': <function sen2agri_temp_feat at 0x7efe9a982ca0>, 'parameters': {'time_start': None, 'time_freq': 10, 'w': 2, 'delta': 0.05, 'tsoil': 0.2}, 'bands': ['ndvi'], 'names': ['maxdif', 'mindif', 'difminmax', 'peak', 'lengthpeak', 'areapeak', 'ascarea', 'asclength', 'ascratio', 'descarea', 'desclength', 'descratio', 'soil1', 'soil2']}}, 'SAR': {'percentile_iqr': {'function': <function percentile_iqr at 0x7efed6d0d670>, 'parameters': {'q': [10, 50, 90], 'iqr': [25, 75]}, 'names': ['p10', 'p50', 'p90', 'iqr']}}}, 'rsi_meta': {'OPTICAL': {'rgbBR': {'bands': ['B02', 'B03', 'B04'], 'range': [0, 1], 'clamp_range': False, 'scale': 1, 'native_res': 10, 'func': <function calc_rgbBR at 0x7efe9a3d9ee0>}}}, 'ignore_def_feat': {'OPTICAL': True, 'SAR': True}, 'gddnormalization': False}
2022-12-01 14:22:58.237 | INFO     | 139633939470144:worldcereal.worldcereal_products:_run_block:244 - save_confidence: True
2022-12-01 14:22:58.237 | INFO     | 139633939470144:worldcereal.worldcereal_products:_run_block:244 - save_features: True
2022-12-01 14:22:58.238 | INFO     | 139633939470144:worldcereal.worldcereal_products:_run_block:244 - localmodels: False
2022-12-01 14:22:58.238 | INFO     | 139633939470144:worldcereal.worldcereal_products:_run_block:244 - segment: False
2022-12-01 14:22:58.238 | INFO     | 139633939470144:worldcereal.worldcereal_products:_run_block:244 - decision_threshold: 0.7
2022-12-01 14:22:58.238 | INFO     | 139633939470144:worldcereal.worldcereal_products:_run_block:244 - filtersettings: {'kernelsize': 3, 'conf_threshold': 0.85}
2022-12-01 14:22:58.238 | INFO     | 139633939470144:worldcereal.worldcereal_products:_run_block:244 - features_dir: /tmp/04QDK_702b02/block_features
2022-12-01 14:22:58.239 | INFO     | 139633939470144:worldcereal.worldcereal_products:_run_block:244 - models: {'annualcropland': 'https://artifactory.vgt.vito.be:443/auxdata-public/worldcereal/models/WorldCerealPixelCatBoost/v700/cropland_detector_WorldCerealPixelCatBoost_v700-realms'}
2022-12-01 14:22:58.239 | INFO     | 139633939470144:worldcereal.worldcereal_products:_run_block:244 - inputs: {'OPTICAL': '/tmp/04QDK_702b02/04QDK_702b02_satio_optical.csv', 'SAR': '/tmp/04QDK_702b02/04QDK_702b02_satio_sar.csv', 'TIR': '/tmp/04QDK_702b02/04QDK_702b02_satio_tir.csv', 'DEM': 's3://ewoc-aux-data/CopDEM_20m', 'METEO': '/tmp/04QDK_702b02/04QDK_702b02_satio_agera5.csv'}
2022-12-01 14:22:58.239 | INFO     | 139633939470144:worldcereal.worldcereal_products:_run_block:244 - yearly_meteo: True
2022-12-01 14:22:58.239 | INFO     | 139633939470144:worldcereal.worldcereal_products:_run_block:244 - use_existing_features: True
2022-12-01 14:22:58.239 | INFO     | 139633939470144:worldcereal.worldcereal_products:_run_block:244 - aez: 20088
2022-12-01 14:22:58.239 | INFO     | 139633939470144:worldcereal.worldcereal_products:_run_block:244 - realm: 8
2022-12-01 14:22:58.239 | INFO     | 139633939470144:worldcereal.worldcereal_products:_run_block:244 - start_date: 2020-09-17
2022-12-01 14:22:58.239 | INFO     | 139633939470144:worldcereal.worldcereal_products:_run_block:244 - end_date: 2021-09-17
2022-12-01 14:22:58.240 | INFO     | 139633939470144:worldcereal.worldcereal_products:get_worldcereal_collections:142 - Initializing collections ...
2022-12-01 14:23:02.292 | INFO     | 139633939470144:worldcereal.worldcereal_products:get_worldcereal_collections:171 - Using s3 bucket collection for `OPTICAL` ...
2022-12-01 14:23:02.305 | INFO     | 139633939470144:worldcereal.worldcereal_products:get_worldcereal_collections:171 - Using s3 bucket collection for `SAR` ...
2022-12-01 14:23:02.316 | INFO     | 139633939470144:worldcereal.worldcereal_products:get_worldcereal_collections:171 - Using s3 bucket collection for `METEO` ...
2022-12-01 14:23:02.316 | INFO     | 139633939470144:worldcereal.worldcereal_products:get_worldcereal_collections:171 - Using s3 bucket collection for `DEM` ...
2022-12-01 14:23:02.478 | INFO     | 139633939470144:worldcereal.utils:_get_realm_model_fromurl:229 - Found realm model: https://artifactory.vgt.vito.be:443/auxdata-public/worldcereal/models/WorldCerealPixelCatBoost/v700/cropland_detector_WorldCerealPixelCatBoost_v700-realms/Realm_8/config.json

...

2022-12-01 14:23:21.355 | WARNING  | 139633939470144:worldcereal.worldcereal_products:_log_run_block:466 - Less than 3 off-swath acquisitions found -> safe to skip block
2022-12-01 14:23:21.657 | SUCCESS  | 139633939470144:satio.utils.logs:wrapper_exitlog:84 - Processing of 04QDK_004 completed in 0.39 minutes.
2022-12-01 14:23:21.658 | SUCCESS  | 139633939470144:worldcereal.worldcereal_products:run_tile:530 - Finished!
2022-12-01 14:23:21.658 | INFO     | 139633939470144:ewoc_classif.classif:process_blocks:130 - Skipped block, return code: 1
Uploaded 0 files to bucket | placeholder
2022-12-01 14:23:21.658 | INFO     | 139633939470144:ewoc_classif.classif:run_classif:360 - Cleaning the output folder /tmp/04QDK_702b02

Overview of missing blocks :
blocs_04QDK

Overview of Kauai island (in Hawaï)
OSM_04QDK
@romain-bdc romain-bdc changed the title Global cropmap production V2 - Warning : less than 3 off-swath acquisition Global cropmap production V2 - Warning : less than 3 off-swath acquisitions Dec 1, 2022
@romain-bdc
Copy link
Author

romain-bdc commented Dec 6, 2022 •
edited
Loading

This issue on cropmap concerns the tiles list above, generally it missing between 3 and 15 blocks on a corner of the tile covering water (those tile are on shorelines) :
04QDK_004, 04QEJ_110, 04QGJ_007, 05QKA_007, 05QKC_008, 14PNC_110, 14PPC_110, 16PBV_110, 21NZG_002, 22NBM_005, 24KWF_076, 24MXA_11, 27RYL_100, 38JMS_101, 38JNS_110, 40KCB_111, 42QXH_111, 48MTU_111, 48MXS_115, 48MYS_111, 49LDL_101, 49LEL_110, 50HLG_110, 51PVS_010, 51PVT_109, 53MPV_20, 55GDM_115, 59HQB_005, 60HTG_010, 60HXD_005

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@romain-bdc