Blue/Green Cache

[chubei]

Blue/Green Cache

Between restarts, many bad things can happen. Ordering from good to bad:

1. Whole DAG is consistent and source can start from checkpoint: This is the best case, we just continue running.
2. Sourceless DAG is consistent and source can start from checkpoint: We can continue running with help of the history in sources.
3. DAG is inconsistent and source can start from checkpoint: We can populate the processors with the help of the history in sources and reduce to case 2.
4. Source can not start from checkpoint: We have no choice but to wipe all source history and processor data. This is the case where blur/green cache comes into play.

Note that cases 2 and 3 can be treated as case 4. We just ignore whether the sources can start from checkpoint. But case 4 cannot be treated as case 2 or 3. So if we want to have a MVP checkpointing solution, case 4 should be prioritized.

Aliasing

To implement blue/green cache, we build upon the concept of cache aliasing.

Every Cache (be it RoCache or RwCache) has a unique name, and we maintain a look up table from cache name aliases to the real names.

Endpoint as an alias

In app, currently we create or open the cache with the endpoint name. This will be changed to:

• Check if the cache alias with the endpoint's name exists. If it does, open corresponding cache.
• If it does not, create a new cache, and create a new cache alias with the endpoint's name, which points to the newly created cache.

In API, currently we open the cache with the endpoint name. This will be changed. The API server will find the cache alias with the endpoint's name, and open the corresponding cache.

AliasRedirected event

When the pointee of a cache alias is changed, the app sends an AliasRedirected message to all API servers through internal server.

Upon receiving the message, API servers switch to the cache with the name of the new pointee.

Blur/Green cache

RoCache and RwCache

We make a minor modification to RoCache and RwCache.

Upon commit, we require RwCache to store the commit checkpoint and the instant of the commit, and the RoCache should be able to retrieve that information.

Pipeline

There will be several back and forth communications between pipeline nodes, all happening before execution.

First checkpoints of the nodes are read and consistency is calculated. If the whole DAG is consistent, DAG asks each source if they can start from the checkpoint. If they all can, we are in case 1. The checkpoint information is passed to the SinkFactory's build method.

If the DAG is inconsistent or any of the source can not start from the checkpoint, ignoring case 2 and 3, we're in case 4. The fact that we are starting from scratch is also passed to the SinkFactory's build method.

Sink

In build, Sink compares the pipeline checkpoint with the cache's checkpoint. If they are not the same, Sink creates a new RwCache and writes to it.

Upon receiving the SnapshottingDone message, the endpoint's alias is redirected to the cache that's being written, and AliasRedirected message is sent to API servers.

@getdozer/dozer-dev Please check this and let me know what you think.

[mediuminvader]

Looks good to me

[karolisg]

Looks good to me

[v3g42]

@chubei As discussed, I think this should go hand in hand with migration behavior.

• If you start/ stop an application
• If the source field definitions changed(adding/deleting columns or the source table)
• Changing SQL in dozer-config that produces different schema.

[chubei]

Migration is beyond the scope of this discussion. If we want to design migration together, let's have a call. I have questions about what the experience should be like and the process of determining pipeline node handles.

[v3g42]

Migration is a beast by itself I agree. But migration will definitely need something similar for managing data on the cases I mentioned above.

ElasticSearch supports Aliases and many of the application deployments would do something using this.
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-aliases.html

IMO, we need a higher level method to push a new version of a cache on a different path and a different enpoint. And We should implement all features that require changes using a blue/green or an alias.

We can have a call to discuss if it helps.

[v3g42]

New version looks great. Thanks @chubei