feat(import): simpler import + state restore logic #625
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
floryst
force-pushed
the
simplified-state-load-restore
branch
from
1849857 to
382c45e
Compare
floryst
force-pushed
the
streaming-base
branch
2 times, most recently
from
c7c1a32 to
19e8af5
Compare
The pipeline code has been removed in favor of a simpler chain-of-responsibility approach, using `evaluateChain` and `asyncSelect`. `evaluateChain` is responsible for evaluating a data source against a chain of import handlers until one of them returns a new data source. To keep processing a data source like how the old pipeline code supported nested executions, `evaluateChain` is invoked inside a loop for every data source. `asyncSelect` is used to drive the loop execution, seleting `evaluateChain` promises whenever they are done. The state schema is updated to generically operate on serialized data sources. Instead of special-casing for remote files, the serialized DataSource type encodes this state.
floryst
force-pushed
the
simplified-state-load-restore
branch
from
382c45e to
f59f77d
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit contains two changes that probably can be split up into two separate PRs that can be applied to the base branch. I'm putting this up as a draft for now so @PaulHax can get a sense of the changes.
The pipeline code has been removed in favor of a simpler chain-of-responsibility approach, using
evaluateChainandasyncSelect.evaluateChainis responsible for evaluating a data source against a chain of import handlers until one of them returns a new data source.To keep processing a data source like how the old pipeline code supported nested executions,
evaluateChainis invoked inside a loop for every data source.asyncSelectis used to drive the loop execution, seletingevaluateChainpromises whenever they are done.The state schema is updated to generically operate on serialized data sources. Instead of special-casing for remote files, the serialized DataSource type encodes this state.