Edge-db usecase

[tarasglek]

I was thinking about databases suitable for edge runtimes like Deno, cloudflare workers. The biggest problem there is any sort of database access completely negates benefits of edge compute.

Projects like turso offer in-memory replication, but that doesn't work with short duration of edge handlers.

I think append-only parquet in style of edge-db is actually the best solution because:

1. Can use webcache or other cache apis to bring relevant subsets of data to worker https://deno.com/blog/deploy-cache-api
2. icedb log avoids s3 list operations
3. icedb log can also be cached locally if one uses something like nats to publish writes, then broadcast log changes

So even though columnar isn't the best format for small reads like (is user logged in), the fact that it can run in-memory in-process on local data, completely makes up for it. One also gets mad read scalability, cost, etc.

The only way to get higher performance is to use a proper read-replica OLAP type architecture, which is gonna be operationally way more complex.

One can push this notion of edge all the way down the webpage JS(and web cache api there) to provide a competitive solution to dexie-style react hooks via https://github.com/danthegoodman1/frontlink type stuff.

I think one merely needs to integrate icedb s3-proxy with nats-sub directly into a duckdb extension to enable this.

PS. been following icedb for a while, awesome work.

[danthegoodman1]

I don't think parquet is actually the "best" format anymore (at least standalone), lancedb has shown you can optimize further!

Good spot on the log notification changes, you can actually do this with native AWS flavor S3 features through their bucket notifications

I'm going to convert to discussion, since there doesn't seem to be anything specifically actionable yet

[tarasglek]

what format is lancedb using? I think it's nice to have something standard even if suboptimal.

I think s3 notifications would be not be sufficient because one probably wants 2 notifications like modification-in-progress, modification-done