feat(promiseStore): promiseStore

[code-forger]

💡 Feature request: Holocron PromiseStore

Holocron comes with a sophistocated DataStore, The DataStore is pluggable, shared across the entire system, and scoped to each request in the server.

At a high level, the DataStore is used by:

• holocron itself as a ModuleStatusStore
• modules as a ModuleStore
• fetchye and iguazu as a CacheStore (although technically these stores are registered by the root module)
• One App as a ConfigStore, an IntlStore, a RedirectionStore, and a RenderingStore. This looks a little like this: (organised by domain and type, not structure)

graph TD;
    DataStore-->holocron;
    holocron-->ModuleStatusStore;
    DataStore-->root-module;
    root-module-->ModuleStore;
    ModuleStore-->fetchye;
    ModuleStore-->iguazu;
    fetchye-->CacheStore;
    iguazu-->CacheStore;
    DataStore-->child-module-1;
    child-module-1-->ModuleStore;
    DataStore-->child-module-2;
    child-module-2-->ModuleStore;
    DataStore-->OneApp;
    OneApp-->ConfigStore;
    OneApp-->IntlStore;
    OneApp-->RedirectionStore;
    OneApp-->RenderingStore;

Loading

This system has allowed Holocron, One App, fetchye etc. To solve many problems when it comes to Two Phase SSR, Client Hydration, Api De-duplication... The list goes on.

However, there is an underlying technology that prevents this versatile store being used for Promise's: transmitJS

The DataStore is serialised and sent to the client, where it is deserialised. Promises cannot be meaningfully serialised, so cannot be put in the DataStore.

So why do we need a Promise Store?

A PromiseStore would serve the same purpose as the DataStore, but for promises: To share promises across the system, scoped to the individual request, and even send those promises to the client.

In the immediacy, the promise store would serve as a solution to this bug in fetchye: americanexpress/fetchye#93, which requires Promises be shared across the module boundary.

Additionally, The up coming React Streaming feature in One App requires not only to share promises between modules, but then to be able to send these promises to the client, and hydrate them in such a way they can then be resolved at a later date. ( I mentioned above that Promises cannot be serialised, and this is true, but React Streaming contains a way of passing data from the server to the client, the resolving the client promises with that data, which makes it look like the promise was resolved across the server/client boundary, this would be integrated with the PromiseStore if streaming is enabled ).

Since it is Holocron's responsibility to configure, maintain, serialize, and hydrate the DataStore. I beleive it should also be Holocrons reponsibility to do the same with a hypothetical PromiseStore

So how would this work.

In short, a new object will be attached to extraThunkArguments, that serves as a promise store.

New Thunks and Hooks will be added for accessing the promised from this object, in an abstract enough way that we can change the underlying fabric of the PromiseStore freely in the future.

This PR serves as a reference implementation of whats needed, Now is a good time to jump to the diff, have a read of the code, then come back up here and read my predicted Questions and Answers.

Note: This is not a final implementation. It explicitly lacks unit tests to keep the diff focussed, and may need tweaks when integrating this change into One App and fetchye.

Some questions you might have while reading the PR:

How would Fetchye use this?

The makeServerFetchye function would be extended to be optionally passed a PromiseStore (any object with a storePromise and getPromise function on it, non-one-app users can bring their own, as they do the cache).

in makeOneServerFetchye, it would pass the PromiseStore from holocron into makeServerFetchye.

If there is a PromiseStore passed into makeServerFetchye it will first check for the existence of a promise in the store for the computed cache key. If it exists, it would await it, then return the coerced fields like normal.

This would mean that the second call to the same data would 'inherit' the promise from the first, and act correctly.

Why attach the PromiseStore to the extraThunkArgs?

• Its an existing solution for sharing something 'System Wide'
  • Although not currently used for a whole lot, the extraThunkArgs is Holocron's way of sharing "things" system wide, such as the fetchClient and the rebuildReducer function. We rarely expect our consumers (Module Code Authors) to interact with these extraThunkArgs, instead it serves as an internal transport for "things" that we need anywhere.
• Its scoped to the request
  • This means our promises are scoped to a single request in the server for free.

Why no Hook for storing a promise

This PromiseStore is currently focussed on solving two problems. One Now (the fetchye bug) and one in the future (React Streaming)

Neither of these require the ability to store a promise at request time. But always want to store promises at 'LoadModuleData' time.

Any more questions?

Just ask in a comment :)

Alternatives Considered

Both the fetchye bug, and the Streaming system have considered alternatives to this solution, and infact streaming is working partially without this feature. However both cannot overcome all issues they face without some iteration of a 'System Wide' PromiseStore

Don't attach to extraThunkArgs

I considered this at length, however ultimately to satisfy both System Wide and Scoped to a Single Request, we would simply need to re-engineer the dispatch system and attach the PromiseStore to that. Which adds needless complexity.

Just actually store the Promises in the Redux Store, and handle the serialisation separately.

This was considered (and actually implemented for streaming), however its quite messy, and leaves us with far more code to maintain than the above does.

Futhermore, the Holocron DataStore is immutable and Promises are not an immutable data type, therefore Promises don't belong in the DataStore for more than one reason.

[code-forger]

Expose only the helper functions, to maintain abstraction for future underlying changes

[code-forger]

Attach the PromiseStore to the extraThunkArgs

[10xLaCroixDrinker]

Can we make is private?

[PixnBits]

and maybe a Map? if not a null-prototype object (Object.create(null))

[code-forger]

Addressed both in cbe1f89

[10xLaCroixDrinker]

Suggested change

	store = (domain, key, promise) => {
	set = (domain, key, promise) => {

[code-forger]

I thought about this, I guess I then didn't articulate this, in code or description.

I was thinking about having the store function throw, or silently no-op if the promise already existed. There is no reason to even overwrite a promise, so a collision would indicate something has gone wrong.

Because of this, I didn't like set because i would expect set to both store and update depending if the promise already existed.

[10xLaCroixDrinker]

Can we add a delete?

[code-forger]

I cant see a reason to delete a promise.

The purpose is to share promises across modules a LoadModuleData time.

At what point could you delete one of these promises? a single loadModuleData function does not know where it is executing, it does not know if the promise its just about to delete is still needed by some module that is going to be 'composed' in after the current moment.

So i cant think of a usecase where deleting a promise would be desirable

[10xLaCroixDrinker]

Can they expire? Will this grow infinitely?

[code-forger]

They are scoped to the request. So in the server when the request finishes it will be garbage collected.

In the browser, it will only grow as much loadModuleData is called, and only grow as fast as the fetchye cache grows.

[PixnBits]

and even send those promises to the client

Can we properly serialize everything about a Promise to transport it from the server to the client?
IIRC we hit issues doing that before and so instead remove the Promise: https://github.com/americanexpress/one-app/blob/main/src/universal/utils/transit.js#L46-L51

[code-forger]

and even send those promises to the client

Can we properly serialize everything about a Promise to transport it from the server to the client? IIRC we hit issues doing that before and so instead remove the Promise: https://github.com/americanexpress/one-app/blob/main/src/universal/utils/transit.js#L46-L51

I dont think it perfectly replicates everything about the promise for Streaming. Just the basics.