Add PMTiles support

[pka]

Add support for pmtiles sources. Solves #807

There is still an error, probably when adding the source to the Maputnik sources:

Failed to process sources for 'pmtiles://https://example.com/data/switzerland.pmtiles' TypeError: NetworkError when attempting to fetch resource.
    fetchSources App.tsx:642

@bdon How did you solve this for https://editor.protomaps.com/ ?

[HarelM]

Thanks for taking the time to open this PR.
Please see my comments in the linked issue.
This part of the solution is a simple one, I would be more interested in actually looking at the problem from a UX perspective and trying to solve that and not just add the pmtiles plugin...

[pka]

I saw your comment there, but comparing PMTiles with MBTiles didn't make sense for me. PMTiles can be used as local and remote datasource like mvt files when using an HTTP server with Range Request support.

[HarelM]

But you still need a http server with range support... From my point if view using a tile server or worker is not a big lift.
Moreover, since this is not supported for native it will create a style that might not work, which is not a great experience...

[pka]

There are not many HTTP servers without Range Request support (python -m http.server comes to my mind) and serving a tileset from a very compact single file without an additional tile server makes hosting a lot easier.

Switching the tile source in a style created with Maputnik is a one-liner like jq '.sources.mysource.url="pmtiles:///mytiles.pmtiles" | del(.sources.mysource.tiles)' mystyle.json. The style itself is the same for any platform.

[HarelM]

I could argue that you won't need to run the above script at all and using Martin or tileserver-gl (using docker, or even without) is not that complicated...
Bottom line, I don't feel comfortable with this feature as it is now from a UX perspective (not a technical one).
But if someone else from the maintainers want to approve it, I won't object.

[kevinschaul]

@pka It might be helpful if you could write your full use-case in #807. Where is the pmtiles file you want to reference? Is it on your local machine or hosted somewhere?

For background I also use maputnik + PMTiles but with a separate tool that runs both. If we can all share our use-cases, that could help reveal pain points and potential solutions.

[pka]

@nyurik @lseelenbinder Could you maybe help formulating use cases for styling maps with PMTiles storage?

[bdon]

@bdon How did you solve this for https://editor.protomaps.com/ ?

here is the necessary changes to add PMTiles for vector sources to the UI:

bdon/maputnik@8ac5e36

This works fine with remote sources e.g. you can paste in https://data.source.coop/protomaps/openstreetmap/tiles/v3.pmtiles and then reference the layer called water

However, the "inspect" panel doesn't work yet because it relies on MapLibre internals to fetch TileJSON. The inspect panel will work once protomaps/PMTiles#247 is implemented.

(EDIT: I implemented 247 above, so inspect works now.)

[bdon]

There is a UI issue right now where the pmtiles:// prefix sometimes appears in the modal data sources editor, sometime does not. Ideally the pmtiles:// prefix is totally hidden from the editor, letting you naturally input a HTTP or HTTPS URL of a .pmtiles archive, and the internal Style JSON state has the full pmtiles://https://... string. This requires some changes to the existing patches, PR welcome.

[WebFreak001]

hello what's needed to get this forward? I would really like to use this since my vector tiles are stored in the pmtiles format, which is simpler for me to deploy than mbtiles as well as even being smaller in size.

Hosting a special-made server that can extract and serve the tiles (either trying to obtain replacement mbtiles or using pmtiles serve) seems like a lot of overhead / unnecessary work since in the end with my usage with MapLibre GL JS just uses the simpler pmtiles format anyway.

Right now my setup with pmtiles looks like this:

in a folder with e.g. planet.pmtiles run:

pmtiles serve --public-url='https://maps.localhost:8123' .

to have a compatibility for the tiles JSON, as well as

sudo caddy reverse-proxy --from maps.localhost:8123 --to :8080 --header-down "access-control-allow-origin: *"

to get a local reverse proxy that adds HTTPS with a self-signed and trusted certificate. Then I can use https://maps.localhost:8123/planet.json in maputnik.

If this PR was merged, I wouldn't need pmtiles anymore and could just host a static fileserver (which I'm doing already for my other web developing needs) with CORS enabled and use the URL for it instead.

And in the future this might even allow me to just select a pmtiles file from disk and without needing to upload it anywhere or host any server, it would just read from it in-memory. (e.g. using a HTML file picker and then using Blob.slice instead of range requests)

[nyurik]

I agree this should be added to maputnik. At minimum, we need to rebase this PR to be able to merge it. i don't see any substantial blockers to be honest, as it is indeed a highly useful mode of operation, and we shouldn't require users to host their own tiles anywhere if they have a pmtiles on some s3 or another range-supporting hosting

[pka]

@WebFreak001 thanks for fixing the bug in the modal data sources editor. I've added your commits also to this PR, to keep the full history.

[HarelM]

As mentioned in the other PR, It would be better to have a test to cover this scenario/use case/flow.

[pka]

We could test e.g.

diff --git a/cypress/e2e/modals.cy.ts b/cypress/e2e/modals.cy.ts
index aade75f..5eeb0c6 100644
--- a/cypress/e2e/modals.cy.ts
+++ b/cypress/e2e/modals.cy.ts
@@ -83,6 +83,20 @@ describe("modals", () => {
       });
     });
 
+    it("add new pmtiles source", () => {
+      let sourceId = "pmtilestest";
+      when.setValue("modal:sources.add.source_id", sourceId);
+      when.select("modal:sources.add.source_type", "pmtiles_vector");
+      when.setValue("modal:sources.add.source_url", "https://pmtiles.io/stamen_toner(raster)CC-BY+ODbL_z3.pmtiles");
+      when.click("modal:sources.add.add_source");
+      when.wait(200);
+      then(
+        get.styleFromLocalStorage().then((style) => style.sources[sourceId])
+      ).shouldInclude({
+        url: "pmtiles://https://pmtiles.io/stamen_toner(raster)CC-BY+ODbL_z3.pmtiles",
+      });
+    });
+
     it("add new raster source", () => {
       let sourceId = "rastertest";
       when.setValue("modal:sources.add.source_id", sourceId);

Is there a way to address the source_url input field?

[HarelM]

This still fails on firefox for some reason :-/

[HarelM]

CC:
ja Japanese @keichan34
zh Simplified Chinese @jieme

[HarelM]

Let's wait for the last two translations.

[HarelM]

@pka can you update the package.json file and I'll merge this?
Thanks!

[pka]

Is there a better way to solve this merge conflict? This PR has 3 commits changing package.json and package-lock.json, so a rebase against main is a lot of rather pointless work. You probably have to push the "Mark as resolved" button?

diff package.json.main package.json

56a57
>     "pmtiles": "^4.1.0",

diff package-lock.json.main package-lock.json

42a43
>         "pmtiles": "^4.1.0",
6094a6096,6101
>     "node_modules/fflate": {
>       "version": "0.8.2",
>       "resolved": "https://registry.npmjs.org/fflate/-/fflate-0.8.2.tgz",
>       "integrity": "sha512-cPJU47OaAoCbg0pBvzsgpTPhmhqI5eJjh/JIu8tPj5q+T7iLvW/JAYUqmE7KOB4R1ZyEhzBaIQpQpardBF5z8A==",
>       "license": "MIT"
>     },
10119a10127,10135
>     },
>     "node_modules/pmtiles": {
>       "version": "4.2.1",
>       "resolved": "https://registry.npmjs.org/pmtiles/-/pmtiles-4.2.1.tgz",
>       "integrity": "sha512-Z73aph49f7KpU7JPb+zDWr+62wPv9jF3p+tvvL26/XeECnzUHnQ0nGopXGPYnq+OQXqyaXZPrsNdKxSD+2HlLA==",
>       "license": "BSD-3-Clause",
>       "dependencies": {
>         "fflate": "^0.8.2"
>       }

[HarelM]

Yes, take the package* files from main and install pmtiles package again.

[pka]

That's what I intended to do. Did a second try and this time only the pmtiles lines are conflicting.

[HarelM]

You need to resolve conflicts though for the CI to run and so that we will be able to merge...

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 66.66667% with 12 lines in your changes missing coverage. Please review.

Project coverage is 63.88%. Comparing base (af01346) to head (899f084).
Report is 48 commits behind head on main.

Files with missing lines	Patch %	Lines
src/components/App.tsx	63.63%	8 Missing ⚠️
src/components/ModalSourcesTypeEditor.tsx	60.00%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #938      +/-   ##
==========================================
+ Coverage   59.84%   63.88%   +4.04%     
==========================================
  Files         104      104              
  Lines        3011     5807    +2796     
  Branches      680     1727    +1047     
==========================================
+ Hits         1802     3710    +1908     
- Misses       1209     2096     +887     
- Partials        0        1       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.