POC of WMS (ArcGIS) Layer

[hanbyul-here]

Related Ticket: #943

Description of Changes

Add 'arc' type layer for both /explore page and /exploration (new E&A)

Notes & Questions About Changes

• The 'arc' layer doesn't have a timestamp, but it still returns the domain, so the dashboard considers it a timestamp. - Should we do something about this for demo?
• Do we want a better preview image or any other edits for demo?
• Do we just need a preview url for demo?
• @slesaad worked on analysis result in this pr: DO NOT MERGE: Arcgis imageserver support #887 I reverted the change related to analysis in this pr since data service is not ready for it yet.

Validation / Testing

• link for /explore page: https://deploy-preview-991--veda-ui.netlify.app/data-catalog/esi_4wk/explore?projection=mercator%7C%7C&basemapid=satellite&datetime=2001-01-02T00%3A00%3A00.000Z
• link for new exploration page: https://deploy-preview-991--veda-ui.netlify.app/exploration?datasets=%5B%7B%22id%22%3A%22esi_4wk_layer%22%2C%22settings%22%3A%7B%22isVisible%22%3Atrue%2C%22opacity%22%3A100%2C%22analysisMetrics%22%3A%5B%7B%22id%22%3A%22mean%22%2C%22label%22%3A%22Average%22%2C%22chartLabel%22%3A%22Average%22%2C%22themeColor%22%3A%22infographicB%22%7D%2C%7B%22id%22%3A%22std%22%2C%22label%22%3A%22St+Deviation%22%2C%22chartLabel%22%3A%22St+Deviation%22%2C%22themeColor%22%3A%22infographicD%22%7D%5D%7D%7D%5D&date=2024-05-27T04%3A00%3A00.000Z

[netlify]

✅ Deploy Preview for veda-ui ready!

Name	Link
🔨 Latest commit	42bb3cc
🔍 Latest deploy log	https://app.netlify.com/sites/veda-ui/deploys/666b591517c90a00081b1b61
😎 Deploy Preview	https://deploy-preview-991--veda-ui.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[aboydnw]

Here are my thoughts, curious what @j08lue @smohiudd @anayeaye @slesaad think

The 'arc' layer doesn't have a timestamp, but it still returns the domain, so the dashboard considers it a timestamp. - Should we do something about this for demo?
Unless we think this is a big technical risk that we need to test, I don't think we should worry about that for now. Is it safe to assume in a future iteration these layers will have a timestamp and the dashboard will handle it easily?

Do we want a better preview image or any other edits for a demo?
The image seems reasonable to me...does it match what we see at the source?

Do we just need a preview url for demo?
That seems reasonable to me, unless there is an engineering reason to merge this or create a broader prototype

[slesaad]

@hanbyul-here can we also do a ImageServer collection, that way we'll be able to show support for two kind of arcgis servers - imageserver and mapserver. Here's a sample collection - https://dev.openveda.cloud/api/stac/collections/POWER_901_MONTHLY_RADIATION_UTC (this is the same kind of dataset that I demoed in my PR)

We're also working on refining the time-series for imageserver - will deploy this soon - so we can also add that to the demo

The 'arc' layer doesn't have a timestamp, but it still returns the domain, so the dashboard considers it a timestamp.

jut FYI that not all 'arc' layers are like this, the imageserver collection will have a timestamp

[slesaad]

Here's the statistics proxy for ImageServer - https://hh76k5rqdh.execute-api.us-west-2.amazonaws.com/dev/docs

Use the /statistics endpoint to get statistics for a variable within an ImageServer
The request and response formats are documented here - https://github.com/NASA-IMPACT/arc2stac/tree/dev?tab=readme-ov-file#request

also here are some other arcgis collections to play around with:

• amphibian_richness_habitat30m (mapserver)
• power_901_annual_meterology_utc (imageserver)

[j08lue]

Are you sure ESI_4K does not have time steps?

At least the ArcGIS web viewer (wait for it to load...) shows selectable times in the full range between 2001 and 2024:

https://www.arcgis.com/home/webmap/viewer.html?url=https%3A%2F%2Fgis1.servirglobal.net%2Farcgis%2Frest%2Fservices%2FGlobal%2FESI_4WK%2FMapServer&source=sd

As it switches through time layers, it makes requests against the ArcGIS MapServer API like this one:

https://gis1.servirglobal.net/arcgis/rest/services/Global/ESI_4WK/MapServer/export?dpi=96&transparent=true&format=png32&time=1250553600000%2C1513036800000&layers=show%3A3&bbox=-3373449.2411962026%2C4224528.2473108135%2C2717053.172565022%2C10060648.230939038&bboxSR=102100&imageSR=102100&size=1245%2C1193&f=image

Sadly, the ArcGIS WMS GetCapabilities has no info about available time steps, so we would need to guess these time coordinates... https://gis1.servirglobal.net/arcgis/services/Global/ESI_4WK/MapServer/WMSServer?service=WMS&version=1.3.0&request=GetCapabilities - but we do that already for TiTiler-served periodic datasets.

(Btw, they seem to have ESI_4K both in MapServer and ImageServer: https://gis1.servirglobal.net/arcgis/rest/services/Global)

[j08lue]

+1 for @slesaad's recommendation to include both a MapServer and an ImageServer layer in the POC.

[slesaad]

@j08lue you're right ESI_4K does have timesteps. I was thinking of the other mapserver datasets like this one. We could"guess" the time steps from the ArcGIS web viewer and put that information in the STAC collection manually, although it's not ideal. We'll update the stac collection to have this info!

Edit: actually looks like there is the time step info in the link

<Dimension name="time" units="ISO8601" current="1">2001-01-02T00:00:00.000Z/2024-06-03T00:00:00.000Z/P7D</Dimension>

P7D - 7 days!

[hanbyul-here]

Oh, I didn't know that different data types are integrated from the ArcGIS server. I am a bit confused if they need to be handled differently or not. From the response of the stac endpoint and the fact that /statistics will be available sooner for the ImageServer type dataset, I think that there will be enough differences between these datasets - Can somebody @slesaad or @anayeaye list out how each dataset will behave differently?

The dashboard can 'guess' the timestamp when the dataset is marked as is_periodic, but the esi_4wk data doesn't have it marked as true, and I wonder why. (Also, what would indicate whether the dataset has timestamps or not? Is it still in discuss?)

[j08lue]

P7D - 7 days!

How many languages do you speak, @slesaad?!

[hanbyul-here]

It's been flaky, but this is the link for analysis https://deploy-preview-991--veda-ui.netlify.app/exploration?datasets=%5B%7B%22id%22%3A%22cdd10%22%2C%22settings%22%3A%7B%22isVisible%22%3Atrue%2C%22opacity%22%3A100%2C%22analysisMetrics%22%3A%5B%7B%22id%22%3A%22mean%22%2C%22label%22%3A%22Average%22%2C%22chartLabel%22%3A%22Average%22%2C%22themeColor%22%3A%22infographicB%22%7D%2C%7B%22id%22%3A%22std%22%2C%22label%22%3A%22St+Deviation%22%2C%22chartLabel%22%3A%22St+Deviation%22%2C%22themeColor%22%3A%22infographicD%22%7D%5D%7D%7D%5D&date=2021-12-30T21%3A48%3A00.000Z&aois=%5B%22vwsxS%7Do%7EiF%3FGys%40%7DgyBw%40s%7Em%40_%40crXe%40cx%5Es%40s%7Bh%40fAuvz%40o%40idGqNmdsA%40gw%40f%40w%7C%5EbIaba%40mc%7BCjBm%7B_A%7DEsrjBoJguHl%40%7BixAfLaihBrNofS%7EAygDA%7BkqDuAw%40%3Fic%60ARsePDy%7DdAoVytuAnAao%7CA%7D%40uC%3Fojd%40i%40ewz%40_AgsqABuiE%40opK%3FusaB_%40U%3F_A%7Cjp%40%3FfdI%40rhq%40%40lvP%3FvbAtArn%60AaBp%7Db%40%7BFxyn%40Evc%40gEl%7Ed%40_Jn%7Em%40%7DBdyOkDdgcAg%40%60bOoBbal%40%7B%40%7CqUArf%40kAvd%5C%5CiMlchAQtzA%40lpNHhyp%40D%7CqXBpr%5C%5CMt%7DL%7E%40neGpc%7C%40%7DGx%7DaA_C%60b%5ByZhh%5EcBjkOn%40tk%7DBnJfst%40vKrs_AhNnxkANx%7CjAkK%7EtEGbxKQn_EG%60y_%40s%40hkUA%7Csb%40BnK%3Fvnu%40tAfv%40Ddx%60AfFx%7EXzAxzkAjGxm%40eo%40rdiBXd%7BJ%3F%7EidB%40v%7DhAtAhan%40v%40t%7EqAf%40_Hyz%7DAkFuklA_Dsbt%40G%7BgAjqBosT%22%2C%2258260%22%2Ctrue%5D&selectedForEditing=false&dateRange=2010-05-09T04%3A00%3A00.000Z%7C2022-08-07T04%3A00%3A00.000Z

And I believe all the layers (MapServer, timeless Mapserver, Imageserver) are being visualized - ESI layer has been also flaky: https://deploy-preview-991--veda-ui.netlify.app/exploration?datasets=%5B%7B%22id%22%3A%22esi_4wk_layer%22%2C%22settings%22%3A%7B%22isVisible%22%3Atrue%2C%22opacity%22%3A100%2C%22analysisMetrics%22%3A%5B%7B%22id%22%3A%22mean%22%2C%22label%22%3A%22Average%22%2C%22chartLabel%22%3A%22Average%22%2C%22themeColor%22%3A%22infographicB%22%7D%2C%7B%22id%22%3A%22std%22%2C%22label%22%3A%22St+Deviation%22%2C%22chartLabel%22%3A%22St+Deviation%22%2C%22themeColor%22%3A%22infographicD%22%7D%5D%7D%7D%2C%7B%22id%22%3A%22bcb_amphibian_richness_habitat30m%22%2C%22settings%22%3A%7B%22isVisible%22%3Afalse%2C%22opacity%22%3A100%2C%22analysisMetrics%22%3A%5B%7B%22id%22%3A%22mean%22%2C%22label%22%3A%22Average%22%2C%22chartLabel%22%3A%22Average%22%2C%22themeColor%22%3A%22infographicB%22%7D%2C%7B%22id%22%3A%22std%22%2C%22label%22%3A%22St+Deviation%22%2C%22chartLabel%22%3A%22St+Deviation%22%2C%22themeColor%22%3A%22infographicD%22%7D%5D%7D%7D%2C%7B%22id%22%3A%22cdd10%22%2C%22settings%22%3A%7B%22isVisible%22%3Afalse%2C%22opacity%22%3A100%2C%22analysisMetrics%22%3A%5B%7B%22id%22%3A%22mean%22%2C%22label%22%3A%22Average%22%2C%22chartLabel%22%3A%22Average%22%2C%22themeColor%22%3A%22infographicB%22%7D%2C%7B%22id%22%3A%22std%22%2C%22label%22%3A%22St+Deviation%22%2C%22chartLabel%22%3A%22St+Deviation%22%2C%22themeColor%22%3A%22infographicD%22%7D%5D%7D%7D%5D&date=2021-12-30T21%3A48%3A00.000Z

There is improvement that should be made especially regarding timestamp, but for POC, would this be enough? @slesaad ?

[hanbyul-here]

To detail the improvement we can make in the future re: timestamps

• Non-periodic time stamps & timeDensity: When the timestamp is at the end of the year (e.g., 2020/12/31) but timeDensity is yearly, the stamp for analysis is displayed very close to the next year, but the label shows it as the year before, which is confusing.
• Add Weekly time Density
• Better handling of timeless data

[slesaad]

looks great @hanbyul-here !! thank you so much for working on this.