Fix ES integration test race conditions #6229
Conversation
- Add template creation retry with timeout in factory - Add context to CreateTemplates in writer - Ensure templates are created before returning writer Fixes jaegertracing#6094
|
@yurishkuro will be great if you can review the changes, looking forward for your feedback. |
|
please make sure all commits are signed off |
| case <-ctx.Done(): | ||
| return fmt.Errorf("template creation timeout: %w", ctx.Err()) | ||
| default: | ||
| if err := writer.CreateTemplates(ctx, spanMapping, serviceMapping, indexPrefix); err == nil { |
There was a problem hiding this comment.
I am not seeing how this addresses the original issue. The assumption was that when we call CreateTemplates it may return before the templates are actually created in the ES, and the immediately-after span writes might fail. Doing retries does not address that.
I would also like to understand if that assumption is correct in the first place and why it might happen - is it indeed that ES acks a success before the template changes are actually applied to the db? Or could it be our fault, e.g. by using an async connection? We do something funky with batch inserts that happens in the background, that could be the source of a race.
| return nil, err | ||
| } | ||
| } | ||
| return writer, nil | ||
| } | ||
|
|
||
| func CreateTemplatesWithRetry( |
There was a problem hiding this comment.
| func CreateTemplatesWithRetry( | |
| func createTemplatesWithRetry( |
Fix ES integration test race conditions
Which problem is this PR solving?
Resolves #6094 - ES integration test race condition where spans are written before templates are fully created.
Description of the changes
Added template creation synchronization in factory.go:
Updated writer.go's CreateTemplates:
Sequence is now:
Before: Start -> Create Writer -> Start Template Creation -> Return Writer -> Write Spans (fails since template still creating)
After: Start -> Create Writer -> Create Templates -> Retry if needed -> Templates Ready -> Return Writer -> Write Spans