Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

trollflow2 falsely claims all files generated successfully #156

Open
gerritholl opened this issue Jul 29, 2022 · 0 comments
Open

trollflow2 falsely claims all files generated successfully #156

gerritholl opened this issue Jul 29, 2022 · 0 comments

Comments

@gerritholl
Copy link
Member

Describe the bug

I have three products in my product list. Two are successfully produced and one fails to be produced. In the logfile, trollflow2 reports: "all 2 files produced nominally". It would rather be more useful if it noted that two files were produced nominally and one failed.

To Reproduce

product_list:
  output_dir:
    /data/pytroll/OUTBOXES/generic/
  subscribe_topics:
    - /set/fci/fdss
  reader:
    - fci_l1c_nc
  mask_area: False
  delay_composites: True
  use_tmp_file: False
  staging_zone: /data/pytroll/TMP/staging
  cache_dir: /data/pytroll/TMP/cache
  resampler: null
  radius_of_influence: 60000
  scene_load_kwargs:
    upper_right_corner:
      NE
  areas:
    null:
      areaname: fulldisc
      products:

        'dwd_vis04':
          productname: dwd_vis04
          formats: &config
            - writer: geotiff
              fname_pattern: "{platform_name}-{productname}-{areaname}-{start_time:%Y%m%d_%H%M}-{end_time:%H%M}-devel.tif"
              nbits: 8
              tiled: True
              blockxsize: 512
              blockysize: 512
              compress: lzw
              predictor: 2
              fill_value: 0

        'true_color_raw':
          productname: true_color_raw
          formats: *config

        'convection':
          productname: convection
          formats: *config

workers:
  - fun: !!python/name:trollflow2.plugins.create_scene
  - fun: !!python/name:trollflow2.plugins.load_composites
  - fun: !!python/name:trollflow2.plugins.resample
  - fun: !!python/name:trollflow2.plugins.save_datasets

running with:

satpy_launcher.py -m /tmp/test-message test-pl.yaml

Expected behavior

I expect one composite to fail: the convection composite requires a mixture of 1 km and 2 km channels. I am not resampling in the example playlist. Indeed, Satpy logs a warning message as expected:

[WARNING: 2022-07-29 14:10:56 : satpy.scene] The following datasets were not created and may require resampling to be generated: DataID(name='convection')

I expect that trollflow2 does not claim that all files were produced nominally.

Actual results

trollflow2 claims that all files were produced nominally:

[WARNING: 2022-07-29 14:10:57 : trollflow2.plugins] Skipping convection: "No dataset matching 'DataQuery(name='convection')' found"
/opt/pytroll/pytroll_inst/mambaforge/envs/pytroll-py310/lib/python3.10/site-packages/dask/core.py:119: RuntimeWarning: invalid value encountered in log
  return func(*(_execute_task(a, cache) for a in args))
[DEBUG: 2022-07-29 14:11:14 : trollflow2.launcher] Cleaning up
[INFO: 2022-07-29 14:11:15 : trollflow2.launcher] All 2 files produced nominally in 0:00:35.071999

Environment Info:

  • Trollflow2 Version: 0.13.5.dev30+gb9400c3
  • Satpy Version: 0.36.1.dev192+g15dbf3db
  • Pyresample Version: 1.24.1

Additional context
Add any other context about the problem here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant