kv/kvnemesis: TestKVNemesisMultiNode failed

[cockroach-teamcity]

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on release-23.2.13-rc @ f00c152761838e05049d293198840b12605b545b:

Fatal error:

panic: r74/2: closed timestamp regression 1728738476.047451385,0 -> 1728738475.873928022,0 (cmd "056206e76a1d4c6f" index 14 term 6 (LAI=2 CTS=1728738475.873928022,0))
data: 00056206e76a1d4c6f30016a0a420092010560b002680272aa010aa701000000000000000001000000012f016b12ec123336386263646661643437363136333600ff01000174786e2d0fc8f21c27ca4af2bc9594c58d58a9a70069120408001000180020002800325bfc3a2c74030a480a100fc8f21c27ca4af2bc9594c58d58a9a710011a14ec123336386263646661643437363136333600012a0a08b8969b98c8cfedfe1730e2b6324a0a08b8969b98c8cfedfe1750032a0a08acb9b6f5cbcfedfe177a0020028a010a08d68690e0c0cfedfe17 [recovered]
	panic: r74/2: closed timestamp regression 1728738476.047451385,0 -> 1728738475.873928022,0 (cmd "056206e76a1d4c6f" index 14 term 6 (LAI=2 CTS=1728738475.873928022,0))
data: 00056206e76a1d4c6f30016a0a420092010560b002680272aa010aa701000000000000000001000000012f016b12ec123336386263646661643437363136333600ff01000174786e2d0fc8f21c27ca4af2bc9594c58d58a9a70069120408001000180020002800325bfc3a2c74030a480a100fc8f21c27ca4af2bc9594c58d58a9a710011a14ec123336386263646661643437363136333600012a0a08b8969b98c8cfedfe1730e2b6324a0a08b8969b98c8cfedfe1750032a0a08acb9b6f5cbcfedfe177a0020028a010a08d68690e0c0cfedfe17

Stack:

goroutine 6247357 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x1f07ce5?, {0x6c6e8a0, 0xc012e82db0})
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x65
panic({0x4718280?, 0xc00961e540?})
	GOROOT/src/runtime/panic.go:914 +0x21f
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*rangeAsserter).apply.func1({0x50b2359?, 0xc00d63b350?}, {0xc006fe99c0?, 0x8?, 0xc006fe98d0?})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/asserter.go:250 +0x165
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*rangeAsserter).apply(0xc00adca0a0, 0x2, {0xc017ba2da0, 0x8}, {0x6, 0xe, 0x0, {0xc00adf0620, 0xd5, 0xe0}}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/asserter.go:301 +0x5f5
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Asserter).Apply(0x4f84c20?, 0x41aa3d?, 0x491cca0?, {0xc017ba2da0, 0x8}, {0x6, 0xe, 0x0, {0xc00adf0620, 0xd5, ...}}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/asserter.go:231 +0xa5
github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.kvnemesisTestCfg.testClusterArgs.func7({{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, {0x0, 0x0, 0x0}, {0x0, ...}, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:146 +0x86
github.com/cockroachdb/cockroach/pkg/kv/kvserver.replicaApplyTestingFilters({0xc010d03310?, 0x6c6e8a0?}, 0xc010d4ad20?, 0xc00af76628?, {0xc010d03360?, 0xc002edb270?, 0x0?}, 0xa0?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:117 +0x1f7
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).Stage(0xc010d03310, {0x6c6e8a0, 0xc010d4ad20}, {0x6c8e8a0?, 0xc00af76628})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go:113 +0xe9
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCmdIter({0x6c9dad0?, 0xc010d036b8}, 0xc006fea568)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:192 +0x148
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch(0xc006feaad8, {0x6c6e8a0, 0xc010d4ad20}, {0x6c9dad0, 0xc010d03688})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:284 +0x150
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries(0xc006feaad8, {0x6c6e8a0, 0xc010d4ad20})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:251 +0x95
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1065 +0x1445
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:750 +0x1b0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc00af6f500, 0x6c6e8a0?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 +0x12b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc000f1f490, {0x6c6e8a0, 0xc012e82db0}, {0x6c528c0, 0xc00af6f500}, 0xc008991800)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:418 +0x18d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x6c6e8a0?, 0xc012e82db0?})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x46
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x13a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 5594810
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x415

Log preceding fatal error

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode3639316795
    test_log_scope.go:81: use -show-logs to present logs inline
    kvnemesis_test.go:320: seed: 4267500610607002804
    kvnemesis_test.go:227: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3953684411

Parameters:

• TAGS=bazel,gss,deadlock
• stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-43130

[nvanbenschoten]

This is an instance of #131450. Closing as a duplicate.