Restructure tile server into a library and a binary

[nyurik]

Tile servers have different use cases. Solving it with "one app fits all" may not work for some users. I would like to refactor mbtileserver-rs into a reusable library plus one or more app. This way the simple cases are covered out of the box, while complex ones can use the library to build their own custom solutions.

There are several ways to structure directories. For now we can just have one lib and one binary in the project:

/src/main.rs
/src/lib.rs
/src/<other-files.rs>

This way some app can use mbtileserve lib with different cases:

• use any async lib (not just tokio)
• use any backend storage, not just sqlite
• have their own URL path structure
• have their own request handlers

TBD:

• can crates.io support bin + lib publishing, or do they need to be published to different names?
• what API does the lib need to be useful?
• what should the file structure be?

[maxammann]

I think users of such a library should be free to use any HTTP server library (like hyper) as well as any async-runtime (like tokio). In fact it seems like hyper depends on tokio.

I think the questions is whether there is much left after abstracting away the server-stuff and backend-stuff like SQLite. I think we should continue this discussion by drafting a few use-cases which would require mbtileserver-rs as library. I'll start:

• Embed a tileserver in nginx/apache as a module. This would require everything except the HTTP frontend.

[nyurik]

Thanks, my use cases:

• add extra static serving at certain paths
• on startup, dynamically determine what services will be available
• support tls (this is probably part of Consider migrating to Actix-Web #17)

[maxammann]

• on startup, dynamically determine what services will be available

What exactly do you mean with that? I think enabling/disabling certain features would also be possible though a config.

add extra static serving at certain paths

What should be served on these paths?

[maxammann]

One reason against properly supporting mbtileserver-rs as a library is stability. If we advertise it as a library then we have to keep the API stable. That is maybe not something we want in an early state of the project.

If we just expose an unstable API which should be used internally and by a small group of people, then we can just do it and see how it goes :)

So for me the question is more whether we want to expose a stable API or not.

[ka7eh]

I like the idea of having a library and binary.

The main use case I can think of is using the library in a more generic tile server application. Something like geoserver that uses mbtileserver-rs to handle mbtiles.

I should mention that mbtiles specs mentions SQLite as the backend, so we don't have much flexibility there.

As for API stability, we can check the current implementation against the required stuff in the specs, and if it meets all the criteria, we can call it stable.

[nyurik]

yes, that's pretty much what i had in mind, except that the lib could also have all the web-serving components if needed, e.g. one would just write their own main fn, and make any kind of decisions of what is needed, e.g. they may decide to handle certain query routes and link them to certain backend services.

[ka7eh]

@nyurik re: code structure, I've tried what you've proposed in another tool before, and it worked fine for my simple use case.
You can check it out here (I haven't tested it recently with the newer versions of Rust!): https://github.com/ka7eh/rust-geobuf.

As for naming, crates.io is fine with having a library and binary with the same name under the same package.

[nyurik]

thx for the info, i wasn't sure if its possible to use the same name for both bin and lib

[brendan-ward]

For reference, we tried to do something roughly similar in mbtileserver some years ago, to try and split the CLI & top-level web framework from the logic for handling individual routes, with the intent that it would allow someone to embed the handlers within another app and provide their own web framework of their choosing. To my knowledge, the handlers have not yet been embedded in another app, so we haven't seen validation for that effort, and in particular I don't know if we landed on the right separation of logic. It did add a bit of complexity to what was originally a fairly lightweight application.

It sounds like that might be a little different than what you have in mind here; we used the Go std library route handlers to build our handlers as part of that move (instead of relying on routing from the top-level web framework), but didn't get at the full flexibility of letting applications define their own handlers or URL patterns.

We had a request to support mbtileserver as an nginx unit but never acted on that (lack of capacity / need); not sure how that compares to some of the use cases envisioned here or even really what that would look like from a user's perspective.

Overall, I think it is a good idea to separate out the runtime environment from the code required to provide responses at different routes. However, I'd offer the following suggestions in case it helps here:

• having well-defined use cases for the library functions will help reinforce clean separation of logic, as there is a bit of spillover between the runtime environment and the logic that may be needed on the library side of things (esp. configuration, assigning IDs to services, and dynamic reloading of individual services were all things we struggled with in this separation).
• split out general building blocks into separate crates, like TileJSON in refactor to use tilejson crate #19 ; there is likely value in doing something similar for basic SQLite <=> mbtiles logic (e.g., fetch tile, infer tile type, etc). I don't see an existing crate for such but will have some upcoming needs re: writing tiles so we should strategize around a more general read / write mbtiles crate. Though again this too gets a bit tricky to separate out runtime environments (i.e., connection pools on top of SQLite, unless I'm not thinking about this clearly). My suggestion is to start here first.
• decide what parts of the handlers should be provided by the library vs bring-your-own; given the above building blocks there may just not be a lot of logic left of the handlers that is easily generalizeable. There may be value in providing an API that can resolve tiles from whatever backend (mbtiles SQLite, etc), but you should decide if that belongs here or elsewhere (here, the goals are nice and clear: provide tiles from mbtiles SQLite, which makes it much easier to reason about this project)
• make sure that the out-of-the box experience for firing up mbtileserver-rs is very easy; I think that the (relatively) easy install and startup of mbtileserver was one of our biggest wins over there; would hate to see that suffer out of a desire to generalize the internals.

[nyurik]

@brendan-ward this is awesome, thank you so much!!! I am beginning to think this is slowly morphing into the deprecated but very successful tilelive API ecosystem, which could be a good pattern to follow. I would love to hear your ideas about the #16 as well, thx!

[nyurik]

This has been an amazing discussion, thanks!!! I have started a more general discussion on how different coding efforts can be combined together with some common goals -- see maplibre/maplibre#30 , and it uses some of the ideas from this one. I think we should keep this discussion going for the technical aspects, and the other for the more fundamental goal-setting ideas.