Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Assess bug] RuntimeError: Attempt to create new tiff file #170

Open
JinIgarashi opened this issue Feb 12, 2025 · 3 comments
Open

[Assess bug] RuntimeError: Attempt to create new tiff file #170

JinIgarashi opened this issue Feb 12, 2025 · 3 comments
Labels
bug Something isn't working

Comments

@JinIgarashi
Copy link
Contributor

JinIgarashi commented Feb 12, 2025 •
edited
Loading

In Jupyter notebook, Assess command fails with error of Attempt to create new tiff file

logs

RuntimeError: Attempt to create new tiff file `/data/jin/20240213_training/sample_project/data/population/female_elderly/RWA_female_elderly.tif' failed: /data/jin/20240213_training/sample_project/data/population/female_elderly/RWA_female_elderly.tif: No such file or directory
May be caused by: Deleting /data/jin/20240213_training/sample_project/data/population/female_elderly/RWA_female_elderly.tif failed:
No such file or directory
Full logs 
/data/jin/20240213_training/sample_project /data/jin/20240213_training
2025-02-12 05:38:05,613 - INFO - Assessing component "population" ━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
2025-02-12 05:38:05,618 - INFO - Assessing variable female_elderly━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
2025-02-12 05:38:07,868 - INFO - Going to compute female_elderly from 4 source  -:--:--
files
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Going to process 15 variables ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
2025-02-12 05:38:09,142 - INFO - Evaluating variable female_elderly using zonal  0% -:--:--
stats
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
2025-02-12 05:38:10,810 - INFO - Created 22 records━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
2025-02-12 05:38:11,374 - INFO - Assessing variable elderly_dependency[0m   0% -:--:--
Assessing population          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
2025-02-12 05:38:11,477 - INFO - Assessing variable elderly_total━━━━━━━━━━━━━   7% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
2025-02-12 05:38:11,480 - INFO - Going to compute ━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--
elderly_total=male_elderly+female_elderly
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
2025-02-12 05:38:11,509 - INFO - Assessing variable female_elderly━━━━━━━━━━━━   7% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing female_elderly in RWA ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing female_elderly in RWA ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing female_elderly in RWA ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing female_elderly in RWA ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing female_elderly in RWA ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing female_elderly in RWA ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing female_elderly in RWA ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing female_elderly in RWA ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing female_elderly in RWA ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing female_elderly in RWA ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing female_elderly in RWA ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing female_elderly in RWA ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--
2025-02-12 05:38:12,630 - INFO - Going to compute female_elderly from 4 source  -:--:--
files
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing female_elderly in RWA ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing population            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
Assessing female_elderly in RWA ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   7% -:--:--
Traceback (most recent call last):
  File "/app/.venv/bin/rapida", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/app/.venv/lib/python3.12/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/cbsurge/assess.py", line 140, in assess
    component(progress=progress, variables=variables, force_compute=force_compute)
  File "/app/cbsurge/components/population/__init__.py", line 165, in __call__
    v(year=self.year, country=country, **kwargs)
  File "/app/cbsurge/core.py", line 184, in __call__
    sources = self.resolve(evaluate=True, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/cbsurge/components/population/__init__.py", line 228, in resolve
    var_local_path = var(**kwargs) # assess
                     ^^^^^^^^^^^^^
  File "/app/cbsurge/core.py", line 181, in __call__
    self.compute(**kwargs)
  File "/app/cbsurge/components/population/__init__.py", line 212, in compute
    sources = self.resolve(**kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^
  File "/app/cbsurge/components/population/__init__.py", line 228, in resolve
    var_local_path = var(**kwargs) # assess
                     ^^^^^^^^^^^^^
  File "/app/cbsurge/core.py", line 173, in __call__
    self.compute(**kwargs)
  File "/app/cbsurge/components/population/__init__.py", line 202, in compute
    computed_file = sumup(src_rasters=downloaded_files,dst_raster=local_path, overwrite=overwrite)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/cbsurge/stats/zst.py", line 123, in sumup
    ds = Calc(calc='sum(a,axis=0)', a=files_to_sum, outfile=dst_raster, projectionCheck=True, format='GTiff',
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/osgeo_utils/auxiliary/util.py", line 46, in enable_exceptions_wrapper
    return fun(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/osgeo_utils/gdal_calc.py", line 413, in Calc
    myOut = myOutDrv.Create(
            ^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/osgeo/gdal.py", line 3817, in Create
    return _gdal.Driver_Create(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Attempt to create new tiff file `/data/jin/20240213_training/sample_project/data/population/female_elderly/RWA_female_elderly.tif' failed: /data/jin/20240213_training/sample_project/data/population/female_elderly/RWA_female_elderly.tif: No such file or directory
May be caused by: Deleting /data/jin/20240213_training/sample_project/data/population/female_elderly/RWA_female_elderly.tif failed:
No such file or directory

reproduction

In jupyter notebook

from cbsurge.admin.ocha import fetch_admin

bbox = [28.729248, -2.88698, 31.047363, -0.954565]
BBOX = ",".join(map(str, bbox))
ADMIN_FILE="admin_2_osm_rwa.gpkg"

print(f"Download data for {BBOX}")

! rapida admin osm --bbox={BBOX} --admin_level=2 --clip --dst_path {ADMIN_FILE} --layer-name "admin"
from cbsurge.project import create

NAME="sample_project"

! rapida create --name={NAME} --polygons={ADMIN_FILE}
! pushd {NAME} && rapida assess -c population -f && popd
@JinIgarashi JinIgarashi added the bug Something isn't working label Feb 12, 2025
@JinIgarashi
Copy link
Contributor Author

I am not sure how to reproduce this bug exactly, but I assume

  • if execute assess command without -v option to assess all variables
  • if execute assess command under existing project with data folder.

when I ran the command rapida assess -c population -v male_total -v female_total -f in a blank project folder, the command was done successfully without error.

@JinIgarashi
Copy link
Contributor Author

JinIgarashi commented Feb 12, 2025 •
edited
Loading

I tried small area for only Kigali (Bounding Box (BBOX): [29.934998, -2.088163, 30.326385, -1.763564]) with rapida assess -c population -v total -v male_total -v female_total -f command. it failed with the same error.

both CLI and notebooks has same error.

@JinIgarashi
Copy link
Contributor Author

the bug present with -f option. it works without -f

@iferencik iferencik removed their assignment Feb 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@JinIgarashi @iferencik