Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Gracefully handle missing project files (ref: /media/*.qgs files) #542

Open
1 task done
Raruto opened this issue May 3, 2023 · 3 comments
Open
1 task done

Gracefully handle missing project files (ref: /media/*.qgs files) #542

Raruto opened this issue May 3, 2023 · 3 comments
Assignees
@wlorenzetti
Labels
bug Something isn't working

Comments

@Raruto
Copy link
Contributor

Raruto commented May 3, 2023 •
edited
Loading

Checklist

  • I've searched through the existing issues and this bug has never been reported before

Subject of the issue

Consider sending a more meaningful error message to the client when for some strange reason a user tries to access a project (whose *.qgs) no longer exists on the server.

Right now, you get a generic server error message at client side:

image

Steps to reproduce

  1. create a new project and save it as usual (eg. uploading the buildings_legend-feature-count.qgs file through the integrated file manager)
  2. delete the uploaded file via docker terminal or sftp: /shared-volume/media/projects/buildings_legend-feature-count.qgs
  3. now go back to the project page to get the following:

image

image

Environment

  • g3w-admin: 3.6.dev-20230503103435
  • g3w-client: 3.8.0-alpha.2
  • browser: Firefox 112.0
  • operating system: Windows 11 64-bit

Link to your project

http://localhost:8000/en/admin/qdjango/buildings/projects/update/public-building-management-demo/

Additional info

Docker compose log

here's what you get when you request the following page at client-side: http://localhost:8000/en/map/buildings/qdjango/1/

2023-05-03 14:30:08 WARNING:root:Serializer
2023-05-03 14:30:08 WARNING:root:Before reading project
2023-05-03 14:30:08 ERROR:qgis.server:[Server] - Error when loading project file '/shared-volume/media/projects/buildings_public-building-management-demo.qgs': Unable to open /shared-volume/media/projects/buildings_public-building-management-demo.qgs 
2023-05-03 14:30:08 WARNING:root:Got project: 
2023-05-03 14:30:08 ERROR:qgis.server:[Server] - Error when loading project file '/shared-volume/media/projects/buildings_public-building-management-demo.qgs': Unable to open /shared-volume/media/projects/buildings_public-building-management-demo.qgs 
2023-05-03 14:30:09 WARNING:qdjango.models.projects:Cannot retrieve QgsMapLayer for QDjango layer buildings_2f43dc1d_6725_42d2_a09b_dd446220104a
2023-05-03 14:30:09 Internal Server Error: /api/config/1/qdjango/1
2023-05-03 14:30:09 Traceback (most recent call last):
2023-05-03 14:30:09   File "/code/g3w-admin/qdjango/api/projects/serializers.py", line 480, in to_representation
2023-05-03 14:30:09     readLeaf(l, ret['layerstree'])
2023-05-03 14:30:09   File "/code/g3w-admin/qdjango/api/projects/serializers.py", line 416, in readLeaf
2023-05-03 14:30:09     readLeaf(node, layer['nodes'])
2023-05-03 14:30:09   File "/code/g3w-admin/qdjango/api/projects/serializers.py", line 437, in readLeaf
2023-05-03 14:30:09     layer_serialized_data = layer_serialized.data
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/serializers.py", line 562, in data
2023-05-03 14:30:09     ret = super().data
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/serializers.py", line 260, in data
2023-05-03 14:30:09     self._data = self.to_representation(self.instance)
2023-05-03 14:30:09   File "/code/g3w-admin/qdjango/api/projects/serializers.py", line 743, in to_representation
2023-05-03 14:30:09     qgs_maplayer = self.qgs_project.mapLayers()[instance.qgs_layer_id]
2023-05-03 14:30:09 KeyError: 'buildings_2f43dc1d_6725_42d2_a09b_dd446220104a'
2023-05-03 14:30:09 
2023-05-03 14:30:09 During handling of the above exception, another exception occurred:
2023-05-03 14:30:09 
2023-05-03 14:30:09 Traceback (most recent call last):
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/django/core/handlers/exception.py", line 47, in inner
2023-05-03 14:30:09     response = get_response(request)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/django/core/handlers/base.py", line 181, in _get_response
2023-05-03 14:30:09     response = wrapped_callback(request, *callback_args, **callback_kwargs)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
2023-05-03 14:30:09     return view_func(*args, **kwargs)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/django/views/generic/base.py", line 70, in view
2023-05-03 14:30:09     return self.dispatch(request, *args, **kwargs)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/views.py", line 505, in dispatch
2023-05-03 14:30:09     response = self.handle_exception(exc)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/views.py", line 465, in handle_exception
2023-05-03 14:30:09     self.raise_uncaught_exception(exc)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/views.py", line 476, in raise_uncaught_exception
2023-05-03 14:30:09     raise exc
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/views.py", line 502, in dispatch
2023-05-03 14:30:09     response = handler(request, *args, **kwargs)
2023-05-03 14:30:09   File "/code/g3w-admin/client/api/views.py", line 50, in get
2023-05-03 14:30:09     elif 'onlineresource' in ps.data['metadata'] and \
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/serializers.py", line 562, in data
2023-05-03 14:30:09     ret = super().data
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/serializers.py", line 260, in data
2023-05-03 14:30:09     self._data = self.to_representation(self.instance)
2023-05-03 14:30:09   File "/code/g3w-admin/qdjango/api/projects/serializers.py", line 483, in to_representation
2023-05-03 14:30:09     'Layer %s is missing from QGIS project!' % l['id'])
2023-05-03 14:30:09 KeyError: 'id'
2023-05-03 14:30:09 Internal Server Error: /api/config/1/qdjango/1
2023-05-03 14:30:09 Traceback (most recent call last):
2023-05-03 14:30:09   File "/code/g3w-admin/qdjango/api/projects/serializers.py", line 480, in to_representation
2023-05-03 14:30:09     readLeaf(l, ret['layerstree'])
2023-05-03 14:30:09   File "/code/g3w-admin/qdjango/api/projects/serializers.py", line 416, in readLeaf
2023-05-03 14:30:09     readLeaf(node, layer['nodes'])
2023-05-03 14:30:09   File "/code/g3w-admin/qdjango/api/projects/serializers.py", line 437, in readLeaf
2023-05-03 14:30:09     layer_serialized_data = layer_serialized.data
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/serializers.py", line 562, in data
2023-05-03 14:30:09     ret = super().data
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/serializers.py", line 260, in data
2023-05-03 14:30:09     self._data = self.to_representation(self.instance)
2023-05-03 14:30:09   File "/code/g3w-admin/qdjango/api/projects/serializers.py", line 743, in to_representation
2023-05-03 14:30:09     qgs_maplayer = self.qgs_project.mapLayers()[instance.qgs_layer_id]
2023-05-03 14:30:09 KeyError: 'buildings_2f43dc1d_6725_42d2_a09b_dd446220104a'
2023-05-03 14:30:09 
2023-05-03 14:30:09 During handling of the above exception, another exception occurred:
2023-05-03 14:30:09 
2023-05-03 14:30:09 Traceback (most recent call last):
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/django/core/handlers/exception.py", line 47, in inner
2023-05-03 14:30:09     response = get_response(request)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/django/core/handlers/base.py", line 181, in _get_response
2023-05-03 14:30:09     response = wrapped_callback(request, *callback_args, **callback_kwargs)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
2023-05-03 14:30:09     return view_func(*args, **kwargs)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/django/views/generic/base.py", line 70, in view
2023-05-03 14:30:09     return self.dispatch(request, *args, **kwargs)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/views.py", line 505, in dispatch
2023-05-03 14:30:09     response = self.handle_exception(exc)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/views.py", line 465, in handle_exception
2023-05-03 14:30:09     self.raise_uncaught_exception(exc)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/views.py", line 476, in raise_uncaught_exception
2023-05-03 14:30:09     raise exc
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/views.py", line 502, in dispatch
2023-05-03 14:30:09     response = handler(request, *args, **kwargs)
2023-05-03 14:30:09   File "/code/g3w-admin/client/api/views.py", line 50, in get
2023-05-03 14:30:09     elif 'onlineresource' in ps.data['metadata'] and \
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/serializers.py", line 562, in data
2023-05-03 14:30:09     ret = super().data
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/serializers.py", line 260, in data
2023-05-03 14:30:09     self._data = self.to_representation(self.instance)
2023-05-03 14:30:09   File "/code/g3w-admin/qdjango/api/projects/serializers.py", line 483, in to_representation
2023-05-03 14:30:09     'Layer %s is missing from QGIS project!' % l['id'])
2023-05-03 14:30:09 KeyError: 'id'
2023-05-03 14:30:09 ERROR:django.request:Internal Server Error: /api/config/1/qdjango/1
2023-05-03 14:30:09 Traceback (most recent call last):
2023-05-03 14:30:09   File "/code/g3w-admin/qdjango/api/projects/serializers.py", line 480, in to_representation
2023-05-03 14:30:09     readLeaf(l, ret['layerstree'])
2023-05-03 14:30:09   File "/code/g3w-admin/qdjango/api/projects/serializers.py", line 416, in readLeaf
2023-05-03 14:30:09     readLeaf(node, layer['nodes'])
2023-05-03 14:30:09   File "/code/g3w-admin/qdjango/api/projects/serializers.py", line 437, in readLeaf
2023-05-03 14:30:09     layer_serialized_data = layer_serialized.data
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/serializers.py", line 562, in data
2023-05-03 14:30:09     ret = super().data
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/serializers.py", line 260, in data
2023-05-03 14:30:09     self._data = self.to_representation(self.instance)
2023-05-03 14:30:09   File "/code/g3w-admin/qdjango/api/projects/serializers.py", line 743, in to_representation
2023-05-03 14:30:09     qgs_maplayer = self.qgs_project.mapLayers()[instance.qgs_layer_id]
2023-05-03 14:30:09 KeyError: 'buildings_2f43dc1d_6725_42d2_a09b_dd446220104a'
2023-05-03 14:30:09 
2023-05-03 14:30:09 During handling of the above exception, another exception occurred:
2023-05-03 14:30:09 
2023-05-03 14:30:09 Traceback (most recent call last):
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/django/core/handlers/exception.py", line 47, in inner
2023-05-03 14:30:09     response = get_response(request)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/django/core/handlers/base.py", line 181, in _get_response
2023-05-03 14:30:09     response = wrapped_callback(request, *callback_args, **callback_kwargs)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
2023-05-03 14:30:09     return view_func(*args, **kwargs)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/django/views/generic/base.py", line 70, in view
2023-05-03 14:30:09     return self.dispatch(request, *args, **kwargs)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/views.py", line 505, in dispatch
2023-05-03 14:30:09     response = self.handle_exception(exc)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/views.py", line 465, in handle_exception
2023-05-03 14:30:09     self.raise_uncaught_exception(exc)
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/views.py", line 476, in raise_uncaught_exception
2023-05-03 14:30:09     raise exc
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/views.py", line 502, in dispatch
2023-05-03 14:30:09     response = handler(request, *args, **kwargs)
2023-05-03 14:30:09   File "/code/g3w-admin/client/api/views.py", line 50, in get
2023-05-03 14:30:09     elif 'onlineresource' in ps.data['metadata'] and \
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/serializers.py", line 562, in data
2023-05-03 14:30:09     ret = super().data
2023-05-03 14:30:09   File "/usr/local/lib/python3.10/dist-packages/rest_framework/serializers.py", line 260, in data
2023-05-03 14:30:09     self._data = self.to_representation(self.instance)
2023-05-03 14:30:09   File "/code/g3w-admin/qdjango/api/projects/serializers.py", line 483, in to_representation
2023-05-03 14:30:09     'Layer %s is missing from QGIS project!' % l['id'])
2023-05-03 14:30:09 KeyError: 'id'
@Raruto Raruto added the ux Changes to a current behavior to improve user experience label May 3, 2023
@Raruto
Copy link
Contributor Author

Raruto commented May 3, 2023

Here's what I get instead after clicking on delete link (when trying to upload a new *.qgs file)

2023-05-03 14:56:39 WARNING:root:Could not resolve form field 'delete_url'.
2023-05-03 14:56:39 Traceback (most recent call last):
2023-05-03 14:56:39   File "/usr/local/lib/python3.10/dist-packages/django/forms/forms.py", line 153, in __getitem__
2023-05-03 14:56:39     field = self.fields[name]
2023-05-03 14:56:39 KeyError: 'delete_url'
2023-05-03 14:56:39 
2023-05-03 14:56:39 During handling of the above exception, another exception occurred:
2023-05-03 14:56:39 
2023-05-03 14:56:39 Traceback (most recent call last):
2023-05-03 14:56:39   File "/usr/local/lib/python3.10/dist-packages/crispy_forms/utils.py", line 80, in render_field
2023-05-03 14:56:39     bound_field = form[field]
2023-05-03 14:56:39   File "/usr/local/lib/python3.10/dist-packages/django/forms/forms.py", line 155, in __getitem__
2023-05-03 14:56:39     raise KeyError(
2023-05-03 14:56:39 KeyError: "Key 'delete_url' not found in 'QdjangoProjectForm'. Choices are: authentication_id, authentication_password, authentication_username, autozoom_query, baselayer, context_base_legend, description, editor2_user, editor_user, editor_user_groups, feature_count_wms, form_id, legend_position, multilayer_query, multilayer_querybybbox, multilayer_querybypolygon, propagate_viewers, qgis_file, qgis_file-metadata, qgis_file-uploads, thumbnail, thumbnail-metadata, thumbnail-uploads, title_ur, toc_layers_init_status, toc_tab_default, toc_themes_init_status, upload_url, url_alias, use_map_extent_as_init_extent, viewer_user_groups, viewer_users."

So, now it seems impossible to upload a new file:

image

@Raruto Raruto added bug Something isn't working and removed ux Changes to a current behavior to improve user experience labels May 3, 2023
@wlorenzetti wlorenzetti mentioned this issue May 4, 2023
Merged
@wlorenzetti
Copy link
Member

This is an edge case, because it is highly unlikely that a project will be deleted from the media folder. It can only happen by manually deleting the file from the folder

@Raruto
Copy link
Contributor Author

Raruto commented May 4, 2023

It can only happen by manually deleting the file from the folder

I've inadvertently deleted the /shared-volume/ folder while migrating from v3.5 to v3.6 and then i tried to restore it but, if I remember correctly, in such cases the only way to fix is to manually re-upload / individually save all the projects via the graphical interface (that means, even if you re-upload the /shared-volume/media/projects/ folder via FTP, the system is no longer able to link these files to your qdjango projects).

BTW, then to make it faster I simply performed a database reset (anyway, at least when a file is missing the system shouldn't crash..)

@Raruto Raruto mentioned this issue Nov 24, 2023
Open
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wlorenzetti wlorenzetti

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@wlorenzetti @Raruto