[Dev Notes] Signia vs Materialite

[tantaman]

tldr:

Materialite assumes you’re dealing with collections and makes incremental computation against collections easy.

Signia assumes you’re dealing with values and makes computations with arbitrary numbers of inputs easy.

The Common

• They both create graphs of computation.
  Signia: atoms → signals → computed signals → further computed signals
  Materialite: collections → streams → operators → further operators
• They both re-compute the sections of the graph as needed when a dependency changes. In Signia's case, when an atom changes. In Materialite's case, when a collection changes.
• Both have ways of combining signals/streams together to create aggregate computations.

The Differences

Base Primitive:

1. The Signia primitive is an atom. An atom is a wrapper that can hold any value and signals when its contents is replaced with a new value.
2. Materialite's primitive is a collection. Materialite signals when anything is added or removed from a collection.

One could fake an Atom in Materialite by creating a collection that only contains one value. If you do that, you're really just using Materialite to represent and re-run your computation graph.

Impacts of the primitive choice:

1. Since Materialite knows it is dealing with collections it can provide a full set of incremental collection operations out of the box.
2. Materialite emits a stream of events as a source collection is interacted with (row added, row removed). This is abstracted from the user when using filter/map/reduce/other stream operators but is what is happening under the hood.
3. Since Signia uses Atoms and Atoms don't know of their contents, Signia sends the full value of an atom whenever it is updated. E.g., "atom set to new value X"

In other words:
The default mode of Materialite is streaming incremental updates to collections.
The default mode of Signia is managing and passing full values.

Since the default mode of Materialite is being incremental and passing change events, what does a function do that needs to see a full collection?

This is simple in Signia since each signal just gets the full value.

Signia:

const source = atom('source', [...]);
const mapped = computed('mapped', () => source.map((x) => ...));
const filteredCollection = computed('filter', () => mapped.filter((x) => ...));

In Signia, each step has access to the full collection emitted by the prior step. Default: non incremental.

In Materialite we need to do something different. To get a view of the full collection after a chain of computations, we materialize the stream.

Materialite:

const source = materialite.newCollection([...]);
const filteredStream = source.map((x) => ...).filter((x) => ...);
const filteredCollection = filteredStream.materialize();

In Materialite, each step only has access to a single row of the collection until materialization occurs. Default: incremental.

Aside from materialization, the other option is to create an operator that remembers everything, or some part of, what it has seen. E.g., count works like that -- it remembers the count of add and remove events that have passed by.

For incremental updates, Signia does have an ImmerAtom that holds an [immer collection](https://github.com/immerjs/immer). Given Signia is always passing full values, incrementalism here is done by the signal asking for a diff of the value it received. I need to look more into these diffs in signia. (How expensive are they to keep once you have long compute chains? How often do you run into their "history limit" preventing a diff from being retrieved?). Materialite doesn't need to store any diffs. It only needs to keep around the source collection which can be a history-free mutable structure. Change events are ephemeral and dropped once they reach the end of a pipeline.

Combining Signals

Because Signia defaults to dealing in values, Signia makes it easy to combine signals in arbitrary ways. You can pass any old function to computed and use any number of atoms, or other computed signals, inside.

const firstName = atom('firstName', 'David')
const lastName = atom('lastName', 'Bowie')

const fullName = computed('fullName', () => {
  return `${firstName.value} ${lastName.value}`
})

This is not quite as nice in Materialite. While operating on a single stream is trivial (we can use any function we want in things like map, reduce, filter), having a function operate against many streams at once is more difficult.

To use values from multiple streams in the same function we're limited to join.

stream1
  .join(stream2, keyFn1, keyFn2)
  .map(([stream1_value, stream2_value]) => {
    return `${stream1_value} ${stream2_value}`;
  });

Although this is somewhat of an apples and oranges comparison. In Materialite you're combining incremental streams. In Singia you're combining full values. The latter is a much easier problem. I wonder what it'd look like to try to do incremental composition signals in Signia…

Creating a computation that is composed of many materialized views would be the Materialite version of what Signia is doing. I haven’t thought about implementing a feature like that but I see it could be pretty useful.

Notional example:

const collection1 = stream1.materialize();
const collection2 = stream2.materialize();
const derivedCollection = materialite.computed(() => {
  // do stuff with full access to `collection1` and `collection2`
});

Other - Hoisting

Maybe we can do this in Signia but, since your feedback last week, I've been adding the idea of "hoisting" operators to Materialite.

In other words:

source.after(cursor).take(n)

If the source is something like Replicache, SQLite, or a sorted collection after can be hoisted and run directly in the database (or jump directory to that spot in the sorted collection). In some cases take can be hoisted as well.

This involves propagating backwards and down to the source information about operators that have been attached.