Integrating with SolidJS store

[tslocke]

Following up with a Discord conversation with @aboodman.

A very brief look at how state works in Solid...

<Counter/> component

function Counter(props) {
  const [count, setCount] = createSignal()
  cont timesTen = () => count() * 10
  return <div style={{color: props.color}}>
    <p>Count is {count()}. Ten times count is {timesTen()}</p> 
    <button onClick={() => setCount(count()+1)}>Inc!</button>
  </div>
}  

Unlike React, the function runs once for each <Counter/> in my app. The calls to count() inside the div are "tracked" - Solid keeps track of dependencies such that the call to setCount will result in the text content of the <p> being mutated directly. There is no dom diffing.

Similarly with props. props.color is invoking a getter, which is also tracked. We say props is a "reactive object". Note that because props.color is actually a function call, if you unpack with const {color} = props outside of the div, you break reactivity. Only the code inside JSX is tracked.

This also gives you reactive computeds for free - they are just plain js functions. The JSX calls timesTen, which calls count, so it gets tracked just the same. There is also createMemo(() => heavyComputation(count())) which gives you a computed that runs when (and only when) the state it depends on changes.

Stores

Say I have a bunch of state:

const [a, setA] = createSignal(1)
const [b, setB] = createSignal(2)
const [c, setC] = createSignal(3)

I can instead do...

const [state, set] = createStore({a: 1, b: 2, c: 3})

...and then use state.a and set('a', 100) instead of a() and setA(100)

But more than that, I can have deeply nested objects and they are reactive all the way down.

Read: state.deeply.nested.field
Write: set('deeply', 'nested', 'field', {newObject: "will also become deeply reactive"})

And that's about it! set has a bunch of super useful convenience features, but they don't change the model.

Unidirectional flow

This is just a usage pattern. It's common to create a big store and either have it global, or make it available with Solid's context feature (very similar to context in React). We don't tend to make setStore (a.k.a set in above examples) available, but instead do:

export function myStore() {
  const [store, setStore] = createStore({
    // initial state
  })

  const actions = {
    establishWorldPeace(x, y) {
      // Do stuff and update state with setStore
    },

    findWaldo() {
      // ...
    }
  }

  return [store, actions]
}

The fact that store (read only) and actions are separate is key to helping keep a unidirectional flow in the app.

Replicache

Finally to the point. Here's how I keep my store in sync with replicache.

The state is duplicated - everything is stored in both the Solid store and in Replicache. This is clearly a downside...

The store actions update both - i.e. they call setStore and repli.mutate.whatever(...)

On app startup, I use repli.query and tx.scan to get everything and populate the Solid store. Solid provides batch(() => {...}) so you can do a bunch of updates and only trigger reactions at the end.

The harder part, of course, is staying in sync with other users/tabs.

I have a custom puller and and onSync:

  let patch = null
  repli.puller = async (req) => {
    const result = await origPuller(req)
    if (result.response && !result.response.error) {
      patch = result.response.patch
      await willApplyPull(patch)
    }
    return result
  }

  repli.onSync = async (syncing) => {
    if (!syncing && patch) {
      await didApplyPull(patch)
      patch = null
    }
  }

In the case of del ops, I might need to get at the object before repli deletes it. So willApplyPull just grabs a copy (tx.get(key)) for every del.

The sync of the Solid store happens in didApplyPull. It does everything inside a single read-transaction (query). It does the obvious thing: del - delete object in solid store, put - update object in solid store, clear - reset solid store. But it also needs to take into account the possibility that things have changed locally, due to un-pushed or in-flight mutations:

• del: only delete it from Solid if tx.get(key) returns null (i.e. check if it got deleted and then came back - new objects always have a new ID but deleted IDs can re-appear due to undo)
• put: update the Solid store with the value from tx.get(key), not the value from the put op. If it's gone, do nothing.
• clear: fully reset the Solid store to the current repli state.

As a last detail, in the put handler I do a field by field deep equality test of repli values to Solid values, and only call setStore on changed fields.

Fingers crossed you are not going to tell me this is all fatally, irredeemably flawed 😬 🤣

Tom

[tslocke]

Quick follow-up to add, this is not just about performance.

e.g. Solid's 'surgical' DOM updates are better at preserving UI state, such as keyboard focus, because it's not regenerating anything. Obviously React solves these problems too but I believe (might be wrong) this relies on special knowledge of specific elements like text inputs. Integrating 3rd party stateful UI components is an absolute breeze in Solid.

e.g. I might want to implement a visual highlight when some field is updated by a remote user. I don't want to flash the whole object.

Also ping @aboodman : )

[aboodman]

The store actions update both - i.e. they call setStore and repli.mutate.whatever(...)

This looks scary to me because we have the same state in two different places and updated via two different mechanisms. It seems just like programming being what it is, we should be able to find some path where the two states diverge. Let's try...

My first questions are:

• What happens when watches/subscriptions are fired in response to repli.mutate.whatever? You must ignore them, right? Because the point is to get Solid's surgical updates and watch gives you the whole value again.
• What happens if setStore and repli.mutate.whatever don't do the same thing?

It seems like if setStore does something different than repli.mutate.whatever then optimistic mutations might not be correctly reverted which would be very confusing.

[aboodman]

Let's assume that the code that changes the store does the exact same thing that rep.mutate.whatever does, given same input state. Even then I think we can find a case where they diverge:

rep.mutate.foo() is async and serialized with respect to other accesses to Replicache. It might not run right away. There is nothing that guarantees that some other write to Replicache might not happen first, in which case the mutate call will run with some other starting state.

[tslocke]

What happens when watches/subscriptions are fired in response to repli.mutate.whatever

I don't have any

What happens if setStore and repli.mutate.whatever don't do the same thing?

That would be a bug, so, something bad

It seems like if setStore does something different than repli.mutate.whatever then optimistic mutations might not be correctly reverted

Whenever a pull comes in, for every key that pull touches I fully reset that object in the Solid store to the state in the cache. Is that sufficient to address your concern?

rep.mutate.foo() is async and serialized with respect to other accesses to Replicache. It might not run right away. There is nothing that guarantees that some other write to Replicache might not happen first, in which case the mutate call will run with some other starting state.

The mutations in this system are all simple CRUD operations. None of them depend on the starting state. I can see that if any of them did - an incrementing counter for example - that would break the whole idea. If I need something like that I will keep the state entirely in replicache and use watch/subscribe.

[aboodman]

The mutations in this system are all simple CRUD operations. None of them depend on the starting state. I can see that if any of them did - an incrementing counter for example - that would break the whole idea. If I need something like that I will keep the state entirely in replicache and use watch/subscribe.

The U in CRUD depends on the starting state. A patch update does something different if the entity exists vs if it doesn't.

function patchState(id, changedFields) {
  patchSolidState(id, changedFields);
  rep.mutate.patchReplicache(id, changedFields);
}

It is possible in principle that when patchReplicache runs, the entry identified by id has been deleted. I'm not sure how that interacts with what you do with pull - I still need to think about that.

[aboodman]

This sequence of operations is possible:

1. patchSolidState
2. pull
3. patchReplicacheState

It is very hard for me to reason about what weird things might happen here because it depends on what's in 1/3 and 2.

[aboodman]

The other thing I would be broadly concerned about is:

The sync of the Solid store happens in didApplyPull. It does everything inside a single read-transaction (query).

There is nothing that guarantees that a write can't sneak in between didApplyPull and the read-transaction. I'm not sure if that matters.

[tslocke]

Thanks for these insights - I'm thinking about all of them carefully.

The U in CRUD depends on the starting state. A patch update does something different if the entity exists vs if it doesn't.

In that case the delete trumps the patch, regardless of the order in which things happen. At some point the object will be deleted in the database, and the delete will be synced to all clients on the subsequent pulls. Patches that may happen before or after in an unpredictable way will not change that.

This sequence of operations is possible:
1. patchSolidState
2. pull
3. patchReplicacheState

I agree, that will cause the states to diverge at a point in the algorithm where they are normally in sync, but then there is

4. pull

and things come back into sync.

I think ultimately all this is solved by the Replicache principle that I am still adhering to - that every change gets pulled, even changes that a given client already knows about. The server side database is the source of truth, and any changes to it get pulled.

For a client side state divergence to get stuck, it would have to never reach the database. The only case I can think of is the one you mention above, where a patch gets dropped because the object was deleted, but in that case we want the delete to trump the patch, so we are good.

And it bears repeating - I agree none of this is sound for the general replicache idea, that you can put arbitrary logic in your mutations and it all remains distributed-system-safe. That's actually amazing and I think you should make a much bigger deal about it in your marketing. It's only really sunk in for me in this conversation.

But I'm not relying on that power here - my mutations are just dumb writes.

[aboodman]

That's actually amazing and I think you should make a much bigger deal about it in your marketing.

Thanks, I've gotten that feedback a number of times recently :).

[tslocke]

What happens if setStore and repli.mutate.whatever don't do the same thing?

That would be a bug, so, something bad

Silly me for thinking you would ask a dumb question!

This can indeed happen in my app, if there is a fractional-indexing key collision.

However if I understand correctly, this question of yours was a lead-in to the concern about "optimistic mutations might not be correctly reverted", and I think I'm OK in that regard.

Thanks for your help!