wal: reduce flake in TestConcurrentWritersWithManyRecords

[sumeerbhola]

Informs #4035

[cockroach-teamcity]

This change is 

[sumeerbhola]

Reviewable status: 0 of 1 files reviewed, 1 unresolved discussion (waiting on @jbowens)

---

wal/failover_writer_test.go line 668 at r1 (raw file):

			// switchToNewDir does not synchronously create the new writer, so wait
			// until the writer is created.
			<-logWriterCreated

• Not using this channel was an oversight.
• For some reason it still occasionally flakes.
• I did add some instrumentation and verified that the failures are not actual bugs. It's simply that some writer has managed to make progress despite the expectation all should have their writes blocked. Which results in it popping from the recordQueue and later writers not seeing those records. That is, across the union of all writers, there is no missing data.

[jbowens]

Reviewed all commit messages.
Reviewable status: 0 of 1 files reviewed, 1 unresolved discussion

[sumeerbhola]

TFTR!