Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ccl/changefeedccl: TestChangefeedSchemaChangeAllowBackfill_Legacy failed #136407

Open
cockroach-teamcity opened this issue Nov 29, 2024 · 5 comments
Open

ccl/changefeedccl: TestChangefeedSchemaChangeAllowBackfill_Legacy failed #136407

cockroach-teamcity opened this issue Nov 29, 2024 · 5 comments
Labels
A-kv Anything in KV that doesn't belong in a more specific category. branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Nov 29, 2024 •
edited by cockroach-jira-scripts
Loading

ccl/changefeedccl.TestChangefeedSchemaChangeAllowBackfill_Legacy failed on master @ 7d48198a57f014a8828194b90098699f70f0695a:

  github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run()
      pkg/rpc/context.go:861 +0x26f
  github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.MuxRangeFeed.func3()
      pkg/rpc/context.go:1026 +0x1ae

Goroutine 21739 (running) created at:
  github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).serveImpl()
      pkg/sql/pgwire/server.go:1193 +0xa4f
  github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).ServeConn()
      pkg/sql/pgwire/server.go:957 +0xe64
  github.com/cockroachdb/cockroach/pkg/server.(*systemServerWrapper).serveConn()
      pkg/server/server_controller_sql.go:170 +0x1d7
  github.com/cockroachdb/cockroach/pkg/server.(*serverController).sqlMux()
      pkg/server/server_controller_sql.go:90 +0x50b
  github.com/cockroachdb/cockroach/pkg/server.(*serverController).sqlMux-fm()
      <autogenerated>:1 +0xc4
  github.com/cockroachdb/cockroach/pkg/server.startServeSQL.func1.1()
      pkg/server/server_sql.go:1985 +0x39b
  github.com/cockroachdb/cockroach/pkg/util/netutil.(*TCPServer).ServeWith.func1()
      pkg/util/netutil/net.go:186 +0x1c6
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
      pkg/util/stop/stopper.go:498 +0x338
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.gowrap1()
      pkg/util/stop/stopper.go:499 +0x4f

Goroutine 25811 (running) created at:
  github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.MuxRangeFeed()
      pkg/rpc/context.go:1011 +0x46d
  github.com/cockroachdb/cockroach/pkg/rpc.(*internalClientAdapter).MuxRangeFeed()
      <autogenerated>:1 +0x12b
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*rangefeedMuxer).startNodeMuxRangeFeed()
      pkg/kv/kvclient/kvcoord/dist_sender_mux_rangefeed.go:391 +0x401
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*rangefeedMuxer).establishMuxConnection.func1()
      pkg/kv/kvclient/kvcoord/dist_sender_mux_rangefeed.go:349 +0x84
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*rangefeedMuxer).establishMuxConnection.Group.GoCtx.func2()
      pkg/util/ctxgroup/ctxgroup.go:189 +0x11a
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      external/org_golang_x_sync/errgroup/errgroup.go:78 +0x91
==================
=== RUN   TestChangefeedSchemaChangeAllowBackfill_Legacy/sinkless
    helpers_test.go:886: making server as system tenant
    helpers_test.go:962: making sinkless feed factory
    helpers_test.go:1027: pgURL sinkless SinklessFeedUser
    helpers_test.go:1027: pgURL sinkless root
    changefeed_test.go:2566: using legacy schema changer
    testfeed_test.go:270: sinkless feed creating changefeed: CREATE CHANGEFEED FOR add_column_def WITH updated
    testfeed_test.go:277: closing sinkless feed
    testfeed_test.go:270: sinkless feed creating changefeed: CREATE CHANGEFEED FOR add_col_comp WITH updated
    testfeed_test.go:277: closing sinkless feed
=== RUN   TestChangefeedSchemaChangeAllowBackfill_Legacy/sinkless/add_column_computed

Parameters:

  • attempt=1
  • race=true
  • run=2
  • shard=15
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/cdc

This test on roachdash | Improve this report!

Jira issue: CRDB-45024
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-cdc labels Nov 29, 2024
@blathers-crl blathers-crl bot added the A-cdc Change Data Capture label Nov 29, 2024
@exalate-issue-sync exalate-issue-sync bot added the P-2 Issues/test failures with a fix SLA of 3 months label Dec 2, 2024
@rharding6373
Copy link
Collaborator

There was a data race. Here's the full output:

=== RUN   TestChangefeedSchemaChangeAllowBackfill_Legacy/sinkless/drop_column
==================
WARNING: DATA RACE
Write at 0x00c00aa0e018 by goroutine 21739:
  github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).reset()
      pkg/util/tracing/span.go:679 +0x1d6
  github.com/cockroachdb/cockroach/pkg/util/tracing.(*Tracer).newSpan()
      pkg/util/tracing/tracer.go:1005 +0x1aa
  github.com/cockroachdb/cockroach/pkg/util/tracing.(*Tracer).startSpanGeneric()
      pkg/util/tracing/tracer.go:1226 +0xc7a
  github.com/cockroachdb/cockroach/pkg/util/tracing.(*Tracer).StartSpanCtx()
      pkg/util/tracing/tracer.go:1088 +0x1e4
  github.com/cockroachdb/cockroach/pkg/util/tracing.ChildSpan()
      pkg/util/tracing/tracer.go:1612 +0x1b3
  github.com/cockroachdb/cockroach/pkg/server.setupSpanForIncomingRPC()
      pkg/server/node.go:1950 +0x144
  github.com/cockroachdb/cockroach/pkg/server.(*Node).setupSpanForIncomingRPC()
      pkg/server/node.go:1938 +0x31e
  github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal()
      pkg/server/node.go:1604 +0x32f
  github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch()
      pkg/server/node.go:1851 +0x3fd
  github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1()
      pkg/rpc/context.go:713 +0x74
  github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12()
      pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:92 +0x72b
  github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4()
      pkg/rpc/context.go:824 +0x88
  github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3()
      pkg/rpc/context.go:165 +0xf0
  github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4()
      pkg/rpc/context.go:824 +0x88
  github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor()
      pkg/rpc/auth.go:90 +0x2b6
  github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor-fm()
      <autogenerated>:1 +0xd6
  github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4()
      pkg/rpc/context.go:824 +0x88
  github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1()
      pkg/rpc/context.go:132 +0x6a
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr()
      pkg/util/stop/stopper.go:349 +0x246
  github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1()
      pkg/rpc/context.go:130 +0x150
  github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4()
      pkg/rpc/context.go:824 +0x88
  github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2()
      pkg/rpc/context.go:723 +0x75
  github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func8()
      pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:232 +0x5ed
  github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1()
      pkg/rpc/context.go:908 +0x199
  github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3()
      pkg/rpc/context.go:793 +0x462
  github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch()
      pkg/rpc/context.go:916 +0xf2
  github.com/cockroachdb/cockroach/pkg/rpc.(*internalClientAdapter).Batch()
      <autogenerated>:1 +0x29
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch()
      pkg/kv/kvclient/kvcoord/transport.go:209 +0x230
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext()
      pkg/kv/kvclient/kvcoord/transport.go:187 +0x139
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.raceTransport.SendNext()
      pkg/kv/kvclient/kvcoord/transport_race.go:71 +0x2ec
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*raceTransport).SendNext()
      <autogenerated>:1 +0x6b
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas()
      pkg/kv/kvclient/kvcoord/dist_sender.go:2747 +0x2708
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch()
      pkg/kv/kvclient/kvcoord/dist_sender.go:2203 +0x8e4
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges()
      pkg/kv/kvclient/kvcoord/dist_sender.go:1739 +0x101e
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send()
      pkg/kv/kvclient/kvcoord/dist_sender.go:1258 +0xa99
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnLockGatekeeper).SendLocked()
      pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go:77 +0x26b
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnMetricRecorder).SendLocked()
      pkg/kv/kvclient/kvcoord/txn_interceptor_metric_recorder.go:41 +0x1d9
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).sendLockedWithRefreshAttempts()
      pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:234 +0x272
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).SendLocked()
      pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:162 +0x1d8
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnCommitter).SendLocked()
      pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go:144 +0x95e
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnPipeliner).SendLocked()
      pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go:320 +0x244
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSeqNumAllocator).SendLocked()
      pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go:112 +0x3d8
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnHeartbeater).SendLocked()
      pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go:265 +0x7d6
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send()
      pkg/kv/kvclient/kvcoord/txn_coord_sender.go:534 +0xa76
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender()
      pkg/kv/db.go:1133 +0x181
  github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send()
      pkg/kv/txn.go:1295 +0x37c
  github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send-fm()
      <autogenerated>:1 +0x55
  github.com/cockroachdb/cockroach/pkg/kv.sendAndFill()
      pkg/kv/db.go:965 +0x27d
  github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Run()
      pkg/kv/txn.go:805 +0xa7
  github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv.catalogQuery.query()
      pkg/sql/catalog/internal/catkv/catalog_query.go:54 +0x244
  github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv.catalogReader.GetByNames()
      pkg/sql/catalog/internal/catkv/catalog_reader.go:404 +0x244
  github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv.(*catalogReader).GetByNames()
      <autogenerated>:1 +0x11e
  github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv.(*cachedCatalogReader).GetByNames()
      pkg/sql/catalog/internal/catkv/catalog_reader_cached.go:403 +0x669
  github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).lookupDescriptorID()
      pkg/sql/catalog/descs/collection.go:642 +0x234
  github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.(*Collection).LookupObjectID()
      pkg/sql/catalog/descs/collection.go:590 +0x86
  github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.GetDescriptorCollidingWithObjectName()
      pkg/sql/catalog/descs/helpers.go:46 +0xa9
  github.com/cockroachdb/cockroach/pkg/sql.getSchemaForCreateTable()
      pkg/sql/create_table.go:181 +0xba4
  github.com/cockroachdb/cockroach/pkg/sql.(*createTableNode).startExec()
      pkg/sql/create_table.go:310 +0xad2
  github.com/cockroachdb/cockroach/pkg/sql.startExec.func2()
      pkg/sql/plan.go:525 +0x1bb
  github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal.func1()
      pkg/sql/walk.go:107 +0xa9
  runtime.deferreturn()
      GOROOT/src/runtime/panic.go:602 +0x5d
  github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visit()
      pkg/sql/walk.go:74 +0x135
  github.com/cockroachdb/cockroach/pkg/sql.walkPlan()
      pkg/sql/walk.go:38 +0x164
  github.com/cockroachdb/cockroach/pkg/sql.startExec()
      pkg/sql/plan.go:528 +0x3a
  github.com/cockroachdb/cockroach/pkg/sql.(*planNodeToRowSource).Start()
      pkg/sql/plan_node_to_row_source.go:166 +0x225
  github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Columnarizer).Init()
      pkg/sql/colexec/columnarizer.go:178 +0x19e
  github.com/cockroachdb/cockroach/pkg/sql/colexec.(*invariantsChecker).Init()
      pkg/sql/colexec/invariants_checker.go:62 +0x97
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).init()
      pkg/sql/colflow/stats.go:89 +0x77
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).init-fm()
      <autogenerated>:1 +0x17
  github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError()
      pkg/sql/colexecerror/error.go:162 +0x7c
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Init()
      pkg/sql/colflow/stats.go:98 +0xde
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedStatsCollectorImpl).Init()
      <autogenerated>:1 +0x45
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).Run.(*BatchFlowCoordinator).init.func2()
      pkg/sql/colflow/flow_coordinator.go:236 +0x65
  github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError()
      pkg/sql/colexecerror/error.go:162 +0x7c
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).init()
      pkg/sql/colflow/flow_coordinator.go:235 +0x1ee
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).Run()
      pkg/sql/colflow/flow_coordinator.go:269 +0x1ef
  github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run()
      pkg/sql/colflow/vectorized_flow.go:315 +0x345
  github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run()
      pkg/sql/distsql_running.go:927 +0x18f5
  github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun()
      pkg/sql/distsql_running.go:2020 +0x32e
  github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll.func3()
      pkg/sql/distsql_running.go:1734 +0x1b0
  github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll()
      pkg/sql/distsql_running.go:1737 +0x353
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine()
      pkg/sql/conn_executor_exec.go:2441 +0xbb6
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine()
      pkg/sql/conn_executor_exec.go:1987 +0x22ec
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState()
      pkg/sql/conn_executor_exec.go:1193 +0x9293
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1()
      pkg/sql/conn_executor_exec.go:141 +0x184
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling()
      pkg/sql/conn_executor_exec.go:3457 +0x539
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt()
      pkg/sql/conn_executor_exec.go:140 +0xbdd
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func1()
      pkg/sql/conn_executor.go:2349 +0x94b
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd()
      pkg/sql/conn_executor.go:2354 +0x1392
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run()
      pkg/sql/conn_executor.go:2271 +0x3ea
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run()
      pkg/sql/conn_executor.go:2271 +0x3ea
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState()
      pkg/sql/conn_executor_exec.go:1193 +0x9293
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1()
      pkg/sql/conn_executor_exec.go:141 +0x184
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling()
      pkg/sql/conn_executor_exec.go:3457 +0x539
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt()
      pkg/sql/conn_executor_exec.go:140 +0xbdd
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func1()
      pkg/sql/conn_executor.go:2349 +0x94b
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd()
      pkg/sql/conn_executor.go:2354 +0x1392
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run()
      pkg/sql/conn_executor.go:2271 +0x3ea
  github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run()
      pkg/sql/conn_executor.go:2271 +0x3ea
  github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn()
      pkg/sql/conn_executor.go:965 +0x216
  github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommands()
      pkg/sql/pgwire/conn.go:251 +0x706
  github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).serveImpl.func4()
      pkg/sql/pgwire/server.go:1196 +0x21c
Previous read at 0x00c00aa0e018 by goroutine 25811:
  github.com/cockroachdb/cockroach/pkg/util/tracing.(*spanInner).isNoop()
      pkg/util/tracing/span_inner.go:55 +0x108
  github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).IsNoop()
      pkg/util/tracing/span.go:148 +0x152
  github.com/cockroachdb/cockroach/pkg/util/tracing.maybeWrapCtx()
      pkg/util/tracing/context.go:55 +0xe9
  github.com/cockroachdb/cockroach/pkg/util/tracing.(*Tracer).startSpanGeneric()
      pkg/util/tracing/tracer.go:1168 +0x1229
  github.com/cockroachdb/cockroach/pkg/util/tracing.(*Tracer).StartSpanCtx()
      pkg/util/tracing/tracer.go:1088 +0x1e4
  github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13()
      pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:158 +0x319
  github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run.func1()
      pkg/rpc/context.go:856 +0x105
  github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4()
      pkg/rpc/context.go:174 +0xd6
  github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run.func1()
      pkg/rpc/context.go:859 +0x1e2
  github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor()
      pkg/rpc/auth.go:152 +0x563
  github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor-fm()
      <autogenerated>:1 +0xc9
  github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run.func1()
      pkg/rpc/context.go:859 +0x1e2
  github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1()
      pkg/rpc/context.go:141 +0x65
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr()
      pkg/util/stop/stopper.go:349 +0x246
  github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2()
      pkg/rpc/context.go:140 +0x15b
  github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run.func1()
      pkg/rpc/context.go:859 +0x1e2
  github.com/cockroachdb/cockroach/pkg/rpc.serverStreamInterceptorsChain.run()
      pkg/rpc/context.go:861 +0x26f
  github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.MuxRangeFeed.func3()
      pkg/rpc/context.go:1026 +0x1ae
Goroutine 21739 (running) created at:
  github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).serveImpl()
      pkg/sql/pgwire/server.go:1193 +0xa4f
  github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).ServeConn()
      pkg/sql/pgwire/server.go:957 +0xe64
  github.com/cockroachdb/cockroach/pkg/server.(*systemServerWrapper).serveConn()
      pkg/server/server_controller_sql.go:170 +0x1d7
  github.com/cockroachdb/cockroach/pkg/server.(*serverController).sqlMux()
      pkg/server/server_controller_sql.go:90 +0x50b
  github.com/cockroachdb/cockroach/pkg/server.(*serverController).sqlMux-fm()
      <autogenerated>:1 +0xc4
  github.com/cockroachdb/cockroach/pkg/server.startServeSQL.func1.1()
      pkg/server/server_sql.go:1985 +0x39b
  github.com/cockroachdb/cockroach/pkg/util/netutil.(*TCPServer).ServeWith.func1()
      pkg/util/netutil/net.go:186 +0x1c6
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
      pkg/util/stop/stopper.go:498 +0x338
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.gowrap1()
      pkg/util/stop/stopper.go:499 +0x4f
Goroutine 25811 (running) created at:
  github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.MuxRangeFeed()
      pkg/rpc/context.go:1011 +0x46d
  github.com/cockroachdb/cockroach/pkg/rpc.(*internalClientAdapter).MuxRangeFeed()
      <autogenerated>:1 +0x12b
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*rangefeedMuxer).startNodeMuxRangeFeed()
      pkg/kv/kvclient/kvcoord/dist_sender_mux_rangefeed.go:391 +0x401
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*rangefeedMuxer).establishMuxConnection.func1()
      pkg/kv/kvclient/kvcoord/dist_sender_mux_rangefeed.go:349 +0x84
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*rangefeedMuxer).establishMuxConnection.Group.GoCtx.func2()
      pkg/util/ctxgroup/ctxgroup.go:189 +0x11a
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      external/org_golang_x_sync/errgroup/errgroup.go:78 +0x91
==================

@andyyang890 andyyang890 removed their assignment Dec 19, 2024
@andyyang890 andyyang890 added A-kv Anything in KV that doesn't belong in a more specific category. T-kv KV Team and removed A-cdc Change Data Capture T-cdc labels Dec 19, 2024
@andyyang890
Copy link
Collaborator

Reassigning to KV given it looks like the data race is between two pieces of KV code.

@kvoli
Copy link
Collaborator

kvoli commented Dec 19, 2024

Not triaging, but just calling out for the KV team member who does, this failure pre-dates enabling apply_to_all (#133860) or any rangefeed related changes (#137531).

Seems like a tracing issue, which are always a little hairy to track down for races.

@wenyihu6
Copy link
Contributor

Also noting that this predates metamorphic enablement #136995 of the buffered sender for unit tests. But it's worth reviewing since we have been doing some code refactoring in this area before enabling it.

@nvanbenschoten
Copy link
Member

We have a write here:

WARNING: DATA RACE
Write at 0x00c00aa0e018 by goroutine 21739:
  github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).reset()
      pkg/util/tracing/span.go:679 +0x1d6
  github.com/cockroachdb/cockroach/pkg/util/tracing.(*Tracer).newSpan()
      pkg/util/tracing/tracer.go:1005 +0x1aa

and a read here:

Previous read at 0x00c00aa0e018 by goroutine 25811:
  github.com/cockroachdb/cockroach/pkg/util/tracing.(*spanInner).isNoop()
      pkg/util/tracing/span_inner.go:55 +0x108
  github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).IsNoop()
      pkg/util/tracing/span.go:148 +0x152
  github.com/cockroachdb/cockroach/pkg/util/tracing.maybeWrapCtx()
      pkg/util/tracing/context.go:55 +0xe9
  github.com/cockroachdb/cockroach/pkg/util/tracing.(*Tracer).startSpanGeneric()
      pkg/util/tracing/tracer.go:1168 +0x1229

Span.reset is writing immediately after pulling from a sync.Pool, so there must be a use-after-free bug somewhere on the read-side.

The span on the read side comes from the ctx passed to internalClientAdapter.MuxRangeFeed. One thing that jumps out to me is that we don't see any goroutine waiting on rangefeedMuxer.g.Wait in muxRangeFeed. Where is that? If that were to return early, we might finish the tracing spans in the ctx passed to MuxRangeFeed before those goroutines have exited, permitting a use-after-free.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-kv Anything in KV that doesn't belong in a more specific category. branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@nvanbenschoten @rharding6373 @cockroach-teamcity @kvoli @andyyang890 @wenyihu6