Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kv/kvnemesis: TestKVNemesisMultiNode failed #131187

Closed
cockroach-teamcity opened this issue Sep 23, 2024 · 2 comments
Closed

kv/kvnemesis: TestKVNemesisMultiNode failed #131187

cockroach-teamcity opened this issue Sep 23, 2024 · 2 comments
Labels
A-kv Anything in KV that doesn't belong in a more specific category. branch-release-23.2.12-rc C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-kv KV Team
Milestone
23.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Sep 23, 2024 •
edited by cockroach-jira-scripts
Loading

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on release-23.2.12-rc @ 776f2ceffbdec6b2e46407aff9e004338831a4b8:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode3807369225
    test_log_scope.go:81: use -show-logs to present logs inline
    kvnemesis_test.go:320: seed: 2251276547389259536
    kvnemesis_test.go:227: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis794714804
    kvnemesis.go:177: range 91 (/Table/100/"6a024f57dfdcff20") RANGE_CONSISTENT_STATS_INCORRECT:
        stats: {ContainsEstimates:0 LastUpdateNanos:1727089337875652697 LockAge:0 GCBytesAge:38226 LiveBytes:159 LiveCount:3 KeyBytes:795 KeyCount:19 ValBytes:517 ValCount:33 IntentBytes:0 IntentCount:0 LockBytes:0 LockCount:0 RangeKeyCount:1 RangeKeyBytes:51 RangeValCount:1 RangeValBytes:11 SysBytes:4876 SysCount:22 AbortSpanBytes:1208}
        delta (stats-computed): {ContainsEstimates:0 LastUpdateNanos:1727089337875652697 LockAge:0 GCBytesAge:0 LiveBytes:0 LiveCount:0 KeyBytes:0 KeyCount:0 ValBytes:0 ValCount:0 IntentBytes:0 IntentCount:0 LockBytes:0 LockCount:0 RangeKeyCount:0 RangeKeyBytes:0 RangeValCount:0 RangeValBytes:0 SysBytes:-10 SysCount:0 AbortSpanBytes:0}
    kvnemesis.go:212: failures(verbose): /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis794714804/failures
        repro steps: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis794714804/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis794714804/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis794714804/kvs-scan.txt
    kvnemesis_test.go:348: [0] proposed: 4285
    kvnemesis_test.go:350: [0] reproposed unchanged: 53
    kvnemesis_test.go:352: [0] reproposed with new LAI: 7
    kvnemesis_test.go:348: [1] proposed: 1059
    kvnemesis_test.go:350: [1] reproposed unchanged: 3
    kvnemesis_test.go:352: [1] reproposed with new LAI: 17
    kvnemesis_test.go:348: [2] proposed: 560
    kvnemesis_test.go:350: [2] reproposed unchanged: 4
    kvnemesis_test.go:352: [2] reproposed with new LAI: 4
    kvnemesis_test.go:348: [3] proposed: 379
    kvnemesis_test.go:350: [3] reproposed unchanged: 0
    kvnemesis_test.go:352: [3] reproposed with new LAI: 7
    kvnemesis_test.go:357: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:357
        	            				github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:297
        	Error:      	Should be zero, but was 1
        	Test:       	TestKVNemesisMultiNode
        	Messages:   	kvnemesis detected failures
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode3807369225
--- FAIL: TestKVNemesisMultiNode (83.15s)

Parameters:

  • TAGS=bazel,gss,deadlock
  • stress=true
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-42436
@cockroach-teamcity cockroach-teamcity added branch-release-23.2.12-rc C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team labels Sep 23, 2024
@cockroach-teamcity cockroach-teamcity added this to the 23.2 milestone Sep 23, 2024
@pav-kv pav-kv added A-kv Anything in KV that doesn't belong in a more specific category. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. and removed branch-release-23.2.12-rc release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Sep 26, 2024
@blathers-crl blathers[crl]
Copy link

blathers-crl bot commented Sep 26, 2024

Hi @pav-kv, please add branch-* labels to identify which branch(es) this C-bug affects.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@pav-kv
Copy link
Collaborator

pav-kv commented Sep 26, 2024

Dup of #93896. Closing since it's a known bug.

@pav-kv pav-kv closed this as completed Sep 26, 2024
arulajmani added a commit to arulajmani/cockroach that referenced this issue Sep 26, 2024
@arulajmani
kvnemesis: ignore SysBytes:{,-}10 MVCC stats discrepancy
c8e59fb 
We started ignoring these in 398dfc7. However, that commit was
effectively reverted when we addressed cockroachdb#93896 in 2d855d3. However, like
2d855d3 notes in its description, this was best effort. As such, this
can still cause test flakes (e.g. cockroachdb#131187).

This patch effectively resurrects 398dfc7, but also additionally
quietens `SysBytes:-10` failure mdoes.

Epic: none

Release note: None
@arulajmani arulajmani mentioned this issue Sep 26, 2024
Merged
craig bot pushed a commit that referenced this issue Sep 27, 2024
@arulajmani
Merge #131446
0ea0320 
131446: kvnemesis: ignore SysBytes:{,-}10 MVCC stats discrepancy r=pav-kv a=arulajmani

We started ignoring these in 398dfc7. However, that commit was effectively reverted when we addressed #93896 in 2d855d3. However, like 2d855d3 notes in its description, this was best effort. As such, this can still cause test flakes (e.g. #131187).

This patch effectively resurrects 398dfc7, but also additionally quietens `SysBytes:-10` failure mdoes.

Epic: none

Release note: None

Co-authored-by: Arul Ajmani <[email protected]>
blathers-crl bot pushed a commit that referenced this issue Sep 27, 2024
@arulajmani
kvnemesis: ignore SysBytes:{,-}10 MVCC stats discrepancy
ec0b8b4 
We started ignoring these in 398dfc7. However, that commit was
effectively reverted when we addressed #93896 in 2d855d3. However, like
2d855d3 notes in its description, this was best effort. As such, this
can still cause test flakes (e.g. #131187).

This patch effectively resurrects 398dfc7, but also additionally
quietens `SysBytes:-10` failure mdoes.

Epic: none

Release note: None
@blathers-crl blathers-crl bot mentioned this issue Sep 27, 2024
Merged
blathers-crl bot pushed a commit that referenced this issue Sep 27, 2024
@arulajmani
kvnemesis: ignore SysBytes:{,-}10 MVCC stats discrepancy
53422fa 
We started ignoring these in 398dfc7. However, that commit was
effectively reverted when we addressed #93896 in 2d855d3. However, like
2d855d3 notes in its description, this was best effort. As such, this
can still cause test flakes (e.g. #131187).

This patch effectively resurrects 398dfc7, but also additionally
quietens `SysBytes:-10` failure mdoes.

Epic: none

Release note: None
@blathers-crl blathers-crl bot mentioned this issue Sep 27, 2024
Merged
blathers-crl bot pushed a commit that referenced this issue Sep 27, 2024
@arulajmani
kvnemesis: ignore SysBytes:{,-}10 MVCC stats discrepancy
f746509 
We started ignoring these in 398dfc7. However, that commit was
effectively reverted when we addressed #93896 in 2d855d3. However, like
2d855d3 notes in its description, this was best effort. As such, this
can still cause test flakes (e.g. #131187).

This patch effectively resurrects 398dfc7, but also additionally
quietens `SysBytes:-10` failure mdoes.

Epic: none

Release note: None
@blathers-crl blathers-crl bot mentioned this issue Sep 27, 2024
Merged
cthumuluru-crdb pushed a commit to cthumuluru-crdb/cockroach that referenced this issue Oct 1, 2024
@arulajmani @cthumuluru-crdb
kvnemesis: ignore SysBytes:{,-}10 MVCC stats discrepancy
8393ec0 
We started ignoring these in 398dfc7. However, that commit was
effectively reverted when we addressed cockroachdb#93896 in 2d855d3. However, like
2d855d3 notes in its description, this was best effort. As such, this
can still cause test flakes (e.g. cockroachdb#131187).

This patch effectively resurrects 398dfc7, but also additionally
quietens `SysBytes:-10` failure mdoes.

Epic: none

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-kv Anything in KV that doesn't belong in a more specific category. branch-release-23.2.12-rc C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-kv KV Team
Projects
None yet
Milestone
23.2
Development

No branches or pull requests
2 participants
@pav-kv @cockroach-teamcity