You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Ideally, we would like quickwit to be ready to ingest documents right after it responded to a create index request.
In practise this is not the case for ingest v1.
When we create a new index, if we experience the cooldown experienced in #4416, indexer node won't be informed right away about the addition of the new index.
Even before, that information was sent asynchronously, and there was a race condition, although much less likely to occur.
In Ingest V2, this is not a problem but another problem arises : commit=force is not supported yet.
My hunch:
We can assume ingest v1 to be deprecated in the next version and focus on implementing commit=force in the next version.
We can then
Pointers to implement wait_for:
The persist success response includes the targeted shard id and the expected position.
It is possible to listen for the shard positions to be updated via the EventBroker.
We need to be careful about the race condition here:
the event broker will only let us know about the future modifications. The following logic
1. ingest
2. subscribe to event broker
3. wait for callback
Does not work because the update may in theory and in practise happen right before 1 and 2.
As a fix, we can either query for the shard position service after subscribing to the event broker,
or subscribe to the event broker before 1.
1. subscribe to event broker
2. ingest
3. wait for callback
Ideally, we would like quickwit to be ready to ingest documents right after it responded to a create index request.
In practise this is not the case for ingest v1.
When we create a new index, if we experience the cooldown experienced in #4416, indexer node won't be informed right away about the addition of the new index.
Even before, that information was sent asynchronously, and there was a race condition, although much less likely to occur.
In Ingest V2, this is not a problem but another problem arises :
commit=force
is not supported yet.My hunch:
We can assume ingest v1 to be deprecated in the next version and focus on implementing
commit=force
in the next version.We can then
sleep_after
clause in rest integration test, introduce in Debouncing reschedule. #4416test_commit_modes
integration testThe text was updated successfully, but these errors were encountered: