Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ingest V2 misses commit force #4438

Closed
fulmicoton opened this issue Jan 22, 2024 · 1 comment · Fixed by #5350
Closed

Ingest V2 misses commit force #4438

fulmicoton opened this issue Jan 22, 2024 · 1 comment · Fixed by #5350
Assignees
Labels
bug Something isn't working

Comments

@fulmicoton
Copy link
Contributor

fulmicoton commented Jan 22, 2024

Ideally, we would like quickwit to be ready to ingest documents right after it responded to a create index request.
In practise this is not the case for ingest v1.

When we create a new index, if we experience the cooldown experienced in #4416, indexer node won't be informed right away about the addition of the new index.
Even before, that information was sent asynchronously, and there was a race condition, although much less likely to occur.

In Ingest V2, this is not a problem but another problem arises :
commit=force is not supported yet.

My hunch:
We can assume ingest v1 to be deprecated in the next version and focus on implementing commit=force in the next version.
We can then

  • remove the sleep_after clause in rest integration test, introduce in Debouncing reschedule. #4416
  • enable ingest V2 on the test_commit_modes integration test
@fulmicoton fulmicoton added the bug Something isn't working label Jan 22, 2024
@fulmicoton fulmicoton changed the title Complicated regression Ingest V2 misses commit force Jan 22, 2024
@rdettai rdettai linked a pull request Aug 27, 2024 that will close this issue
5 tasks
@rdettai rdettai self-assigned this Aug 27, 2024
@fulmicoton
Copy link
Contributor Author

Pointers to implement wait_for:
The persist success response includes the targeted shard id and the expected position.

It is possible to listen for the shard positions to be updated via the EventBroker.
We need to be careful about the race condition here:
the event broker will only let us know about the future modifications. The following logic

1. ingest
2. subscribe to event broker
3. wait for callback

Does not work because the update may in theory and in practise happen right before 1 and 2.

As a fix, we can either query for the shard position service after subscribing to the event broker,
or subscribe to the event broker before 1.

1. subscribe to event broker
2. ingest
3. wait for callback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants