data uploading pipeline #545
Replies: 9 comments 6 replies
-
|
According to the discussion with @iferencik, I updated the image. First Azure Function to upload data can be replaced svelte endpoint. Also, Azure Blob can trigger Azure Function
References
|
Beta Was this translation helpful? Give feedback.
-
Target of uploaded data formatShorlist some drivers, accept ALL GDAL supported Raster
Vector
GeoDatabases
Uploading approaches
Blob container
folder structure can be....
|
Beta Was this translation helpful? Give feedback.
-
|
This is copy from draft PR#1250
The following procedure can be done associated with this table structure data access managementRegarding to access level for datasets, there are two types.
data uploading and processing managementI created
|
Beta Was this translation helpful? Give feedback.
-
|
additional format should be supported?
|
Beta Was this translation helpful? Give feedback.
-
ContextThe data management functionality in GeoHub is going to support users with (1) uploading (2) publishing and (3) managing their geo-spatial data. The main goal here is to break the geo-spatial data silos within UNDP. A valid assumption is that significant quantities of geo-spatial data sits isolated across the organization and users could help solve this issue by publishing this data through GeoHub. While some users might have geo-spatial background and thus know more about the data it is only safe to assume most of users have little or no knowledge at all about GIS data formats. Another assumption is the users will in general posses a proprietary GIS format from ESRI or MapInfo that will hold multiple GIS layers. User journeyOnly authenticated users will be able to upload GIS data to GeoHub.
flowchart LR
A(Authenticated user) -->B(Datasets table) --> C(Upload new)--file-->B(Datasets table)--Dataset-->D(Publish)-->B(Datasets table)
B--Dataset-->E(un-publish)
B--Dataset-->F(Remove)
Data journeyFor the users to be able to manage their own GIS datasets the Geohub must implement functionality on the front-end as well as on the back-end side. The starting point of this process is the user uploading a data file. By convention, user's data will be hosted in a specific Azure blob storage container inside a folder carrying the email address of the user [email protected]. However, this could affect the final path of the server datasets and the Inside this folder three sub-folders named raw, working, and datasets will be created and will serve the ingest process. The raw folder is the target for the upload process and it will hold the original versions of user uploaded data files. These data files will stay here until the owner of an administrator decides to remove it. The second folder working is meant to hold a copy of the uploaded data files temporarily, until they are ingested. The option to keep the copied data files permanently is also a possibility here, however this could have an impact on the size of the data base. The last folder datasets is the folder where all the user's datasets are going to be stored. The GeoHub layers extracted from any given uploaded data file are going to be stored inside this folder in a sub-folder with the same name as the input data file. For example if the user has uploaded a data file to: [email protected]/raw/afile.tif the resulting layers/GeoHub dataset would be located in [email protected]/datasets/afile.tif/band1.tif The path of a geospatial archive/data file is as follows flowchart LR
subgraph FRONTEND
direction LR
A[Upload]--file-->B[userdata/userid/raw]
end
subgraph BACKEND
direction TB
M[userdata/userid/raw]-->D{Sanity Check}
D{Sanity Check} -- Fail --> E[userdata/userid/corrupted] ----> E1{Manual check}
E1{Manual check} --Fail--> E2[DELETE] --> E3[Email]
E1{Manual check} --Pass-->M[userdata/userid/raw]
F[userdata/userid/working] --create folder --> C[userdata/userid/datasets/datafile]
D{Sanity Check} -- Pass --> F[userdata/userid/working] --Check data type-->G{Data type}
G{Data type} --raster--> R[raster ingest] --COG--> R1(userdata/userid/datasets/datafile/layername.tif)
G{Data type} --vector--> V[vector ingest] --PMTILES--> V1(userdata/userid/datasets/datafile/layername.pmtiles)
G{Data type} --CSV--> Z[CSV ingest] --COG/PMTILES--> Z1(userdata/userid/datasets/datafile/layername.tif?pmtiles)
end
FRONTEND --> |BLOB TRIGGER| BACKEND
|
Beta Was this translation helpful? Give feedback.
-
|
My workspace is a section of GeoHub that allows users to manage their data.
every data file becomes a line with buttons to delete the data file and columns to show the file size in MB/GB, uploaded 2.uploading $ingest |
Beta Was this translation helpful? Give feedback.
-
|
The individual user folder structure in the storage account |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
This is the proposed JSON data which the pipeline is going to send to sveltekit websocket server. {
"user": "a85516c81c0b78d3e89d3f00099b8b15",
"url": "https://undpgeohub.blob.core.windows.net/userdata/a85516c81c0b78d3e89d3f00099b8b15/raw/wasac-rwasom-2-data-revised_20230912143433.gpkg",
"progress": 10,
"stage": "downloading"
}
|
Beta Was this translation helpful? Give feedback.
-
|
Message format should be as following. {
"type": "message",
"from": "server",
"group": "datapipeline",
"dataType": "json",
"data": {
"user": "a85516c81c0b78d3e89d3f00099b8b15",
"url": "https://undpgeohub.blob.core.windows.net/userdata/a85516c81c0b78d3e89d3f00099b8b15/raw/wasac-rwasom-2-data-revised_20230912143433.gpkg",
"progress": 10,
"stage": "downloading"
}
} |
Beta Was this translation helpful? Give feedback.
-
|
For cancelation |
Beta Was this translation helpful? Give feedback.
-
|
The message will be sent to sevice bus queue when users upload the dataset, it follows the following message format. each infomation is split by
|
Beta Was this translation helpful? Give feedback.
-
|
Feature set progress state as blob metadata The geohub-data-pipeline communicates with the GeoHub UI through AsurePubSub Websocket service. The pipeline sends messages with the status and progress on every data file. While this keeps the UI nad pipeline in sync if the user refreshed the page while a dataset is being processed all the progress and status is lost. For this purpose everytime a websocket message is sent the stage and progress is set on the raw datafile as metadata. |
Beta Was this translation helpful? Give feedback.
-
|
Feature: Upload multiple datasets to Geohub
|
Beta Was this translation helpful? Give feedback.
-
|
let's close this discussion since it has been implemented |
Beta Was this translation helpful? Give feedback.
-
@iferencik I made an picture how we can establish data uploading pipeline in Azure.
https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-service-bus-output?tabs=in-process%2Cextensionv5&pivots=programming-language-javascript
https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-service-bus-trigger?tabs=in-process%2Cextensionv5&pivots=programming-language-javascript
Beta Was this translation helpful? Give feedback.
All reactions