Skip to content

Commit

Permalink
testcluster: retry force repl scan on startup
Browse files Browse the repository at this point in the history
There have been recent flakes on tests which use a `TestCluster`. These
failures have occurred at startup, before much test specific logic. This
patch mitigates the symptom by retrying on an error, rather than
erroring out. The error looks like:

```
testcluster.go:485: unable to retrieve conf reader: span configs not available
```

It is caused by span configs not being available, which is checked via
grabbing the `ConfReader` before forcing the replica queues to process.

The root cause of this behavior change should still be determined and
this should be considered temporary, although the root cause may be
benign.

informs: #137712
informs: #137762
Release note: None
  • Loading branch information
kvoli committed Dec 19, 2024
1 parent 69d5559 commit 71cdabc
Showing 1 changed file with 3 additions and 1 deletion.
4 changes: 3 additions & 1 deletion pkg/testutils/testcluster/testcluster.go
Original file line number Diff line number Diff line change
Expand Up @@ -1479,7 +1479,9 @@ func (tc *TestCluster) WaitForFullReplication() error {
// Force upreplication. Otherwise, if we rely on the scanner to do it,
// it'll take a while.
if err := s.ForceReplicationScanAndProcess(); err != nil {
return err
log.Infof(context.TODO(), "%v", err)
notReplicated = true
return nil
}
if err := s.ComputeMetrics(context.TODO()); err != nil {
// This can sometimes fail since ComputeMetrics calls
Expand Down

0 comments on commit 71cdabc

Please sign in to comment.