Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Data race in the SDS e2e server suite #10743

Open
timflannagan opened this issue Mar 4, 2025 · 0 comments
Open

Data race in the SDS e2e server suite #10743

timflannagan opened this issue Mar 4, 2025 · 0 comments

Comments

@timflannagan
Copy link
Member

See the following race for more information:

$ go test -failfast -race -count=1  ./internal/sds/pkg/run/...
Running Suite: SDS Server E2E Test Suite - /work/kgateway/internal/sds/pkg/run
==============================================================================
Random Seed: 1741108988

Will run 3 of 3 specs
{"level":"info","ts":"2025-03-04T12:23:08.262-0500","caller":"server/server.go:77","msg":"sds server listening on 127.0.0.1:8236"}
{"level":"info","ts":"2025-03-04T12:23:08.265-0500","caller":"server/server.go:86","msg":"stopping sds server on 127.0.0.1:8236\n"}
•{"level":"info","ts":"2025-03-04T12:23:08.265-0500","caller":"server/server.go:77","msg":"sds server listening on 127.0.0.1:8236"}
{"level":"info","ts":"2025-03-04T12:23:14.291-0500","caller":"server/server.go:130","msg":"Updating SDS config. sdsClient is test-client. Snapshot version is 16694243763138062080"}
{"level":"info","ts":"2025-03-04T12:23:14.292-0500","caller":"run/run.go:79","msg":"watcher started","sslKeyFile":"/tmp/647372046/tls.key-0","sshCertFile":"/tmp/647372046/tls.crt-0","sslCaFile":"/tmp/647372046/ca.crt-0"}
{"level":"warn","ts":"2025-03-04T12:23:14.292-0500","caller":"run/run.go:81","msg":"{error 26 0  no such file or directory}"}
{"level":"warn","ts":"2025-03-04T12:23:14.292-0500","caller":"run/run.go:84","msg":"{error 26 0  no such file or directory}"}
{"level":"warn","ts":"2025-03-04T12:23:14.292-0500","caller":"run/run.go:87","msg":"{error 26 0  no such file or directory}"}
{"level":"info","ts":"2025-03-04T12:23:14.293-0500","caller":"server/server.go:130","msg":"Updating SDS config. sdsClient is test-client. Snapshot version is 969835737182439215"}
{"level":"info","ts":"2025-03-04T12:23:14.293-0500","caller":"run/run.go:79","msg":"watcher started","sslKeyFile":"/tmp/997912853/tls.key-0","sshCertFile":"/tmp/997912853/tls.crt-0","sslCaFile":"/tmp/997912853/ca.crt-0"}
{"level":"info","ts":"2025-03-04T12:23:14.297-0500","caller":"run/run.go:54","msg":"received event","event":"CHMOD         \"/tmp/997912853/tls.key-0\""}
{"level":"info","ts":"2025-03-04T12:23:20.329-0500","caller":"server/server.go:130","msg":"Updating SDS config. sdsClient is test-client. Snapshot version is 6265739243366543658"}
{"level":"info","ts":"2025-03-04T12:23:20.329-0500","caller":"run/run.go:79","msg":"watcher started","sslKeyFile":"/tmp/997912853/tls.key-0","sshCertFile":"/tmp/997912853/tls.crt-0","sslCaFile":"/tmp/997912853/ca.crt-0"}
==================
WARNING: DATA RACE
Write at 0x00c000024600 by goroutine 74:
  runtime.mapaccess2()
      /home/timflannagan/go/pkg/mod/golang.org/[email protected]/src/internal/runtime/maps/runtime_swiss.go:117 +0x2dc
  github.com/fsnotify/fsnotify.(*watches).updatePath()
      /home/timflannagan/go/pkg/mod/github.com/fsnotify/[email protected]/backend_inotify.go:163 +0x1f0
  github.com/fsnotify/fsnotify.(*inotify).register()
      /home/timflannagan/go/pkg/mod/github.com/fsnotify/[email protected]/backend_inotify.go:335 +0x194
  github.com/fsnotify/fsnotify.(*inotify).add()
      /home/timflannagan/go/pkg/mod/github.com/fsnotify/[email protected]/backend_inotify.go:331 +0x100
  github.com/fsnotify/fsnotify.(*inotify).AddWith()
      /home/timflannagan/go/pkg/mod/github.com/fsnotify/[email protected]/backend_inotify.go:296 +0x320
  github.com/fsnotify/fsnotify.(*inotify).Add()
      /home/timflannagan/go/pkg/mod/github.com/fsnotify/[email protected]/backend_inotify.go:253 +0x44
  github.com/fsnotify/fsnotify.(*Watcher).Add()
      /home/timflannagan/go/pkg/mod/github.com/fsnotify/[email protected]/fsnotify.go:313 +0x278
  github.com/kgateway-dev/kgateway/v2/internal/sds/pkg/run.watchFiles()
      /work/kgateway/internal/sds/pkg/run/run.go:80 +0x24c
  github.com/kgateway-dev/kgateway/v2/internal/sds/pkg/run.Run.func1()
      /work/kgateway/internal/sds/pkg/run/run.go:56 +0x394

Previous read at 0x00c000024600 by goroutine 73:
  runtime.mapdelete_faststr()
      /home/timflannagan/go/pkg/mod/golang.org/[email protected]/src/internal/runtime/maps/runtime_faststr_swiss.go:396 +0x8c
  github.com/fsnotify/fsnotify.(*inotify).readEvents()
      /home/timflannagan/go/pkg/mod/github.com/fsnotify/[email protected]/backend_inotify.go:534 +0x8c0
  github.com/fsnotify/fsnotify.newBufferedBackend.gowrap1()
      /home/timflannagan/go/pkg/mod/github.com/fsnotify/[email protected]/backend_inotify.go:195 +0x34

Goroutine 74 (running) created at:
  github.com/kgateway-dev/kgateway/v2/internal/sds/pkg/run.Run()
      /work/kgateway/internal/sds/pkg/run/run.go:49 +0x294
  github.com/kgateway-dev/kgateway/v2/internal/sds/pkg/run_test.init.func1.4.1()
      /work/kgateway/internal/sds/pkg/run/run_e2e_test.go:135 +0x188

Goroutine 73 (running) created at:
  github.com/fsnotify/fsnotify.newBufferedBackend()
      /home/timflannagan/go/pkg/mod/github.com/fsnotify/[email protected]/backend_inotify.go:195 +0x36c
  github.com/fsnotify/fsnotify.newBackend()
      /home/timflannagan/go/pkg/mod/github.com/fsnotify/[email protected]/backend_inotify.go:174 +0x58
  github.com/fsnotify/fsnotify.NewWatcher()
      /home/timflannagan/go/pkg/mod/github.com/fsnotify/[email protected]/fsnotify.go:253 +0x48
  github.com/kgateway-dev/kgateway/v2/internal/sds/pkg/run.Run()
      /work/kgateway/internal/sds/pkg/run/run.go:38 +0xb4
  github.com/kgateway-dev/kgateway/v2/internal/sds/pkg/run_test.init.func1.4.1()
      /work/kgateway/internal/sds/pkg/run/run_e2e_test.go:135 +0x188
==================
{"level":"info","ts":"2025-03-04T12:23:20.332-0500","caller":"run/run.go:54","msg":"received event","event":"REMOVE        \"/tmp/997912853/tls.key-0\""}
{"level":"info","ts":"2025-03-04T12:23:26.365-0500","caller":"server/server.go:130","msg":"Updating SDS config. sdsClient is test-client. Snapshot version is 14893951670674740726"}
{"level":"info","ts":"2025-03-04T12:23:26.365-0500","caller":"run/run.go:79","msg":"watcher started","sslKeyFile":"/tmp/997912853/tls.key-0","sshCertFile":"/tmp/997912853/tls.crt-0","sslCaFile":"/tmp/997912853/ca.crt-0"}
{"level":"info","ts":"2025-03-04T12:23:26.365-0500","caller":"run/run.go:54","msg":"received event","event":"CHMOD         \"/tmp/997912853/tls.key-0\""}
•{"level":"info","ts":"2025-03-04T12:23:27.376-0500","caller":"server/server.go:86","msg":"stopping sds server on 127.0.0.1:8236\n"}
{"level":"info","ts":"2025-03-04T12:23:27.379-0500","caller":"server/server.go:77","msg":"sds server listening on 127.0.0.1:8236"}
{"level":"info","ts":"2025-03-04T12:23:31.900-0500","caller":"server/server.go:130","msg":"Updating SDS config. sdsClient is test-client. Snapshot version is 6730780456972595554"}
{"level":"info","ts":"2025-03-04T12:23:31.901-0500","caller":"run/run.go:79","msg":"watcher started","sslKeyFile":"/tmp/742621040/tls.key-0","sshCertFile":"/tmp/742621040/tls.crt-0","sslCaFile":"/tmp/742621040/ca.crt-0"}
{"level":"info","ts":"2025-03-04T12:23:32.396-0500","caller":"server/server.go:130","msg":"Updating SDS config. sdsClient is test-client. Snapshot version is 16694243763138062080"}
{"level":"info","ts":"2025-03-04T12:23:32.396-0500","caller":"run/run.go:79","msg":"watcher started","sslKeyFile":"/tmp/997912853/tls.key-0","sshCertFile":"/tmp/997912853/tls.crt-0","sslCaFile":"/tmp/997912853/ca.crt-0"}
{"level":"warn","ts":"2025-03-04T12:23:32.396-0500","caller":"run/run.go:81","msg":"{error 26 0  no such file or directory}"}
{"level":"warn","ts":"2025-03-04T12:23:32.396-0500","caller":"run/run.go:84","msg":"{error 26 0  no such file or directory}"}
{"level":"warn","ts":"2025-03-04T12:23:32.396-0500","caller":"run/run.go:87","msg":"{error 26 0  no such file or directory}"}
{"level":"info","ts":"2025-03-04T12:23:32.396-0500","caller":"run/run.go:54","msg":"received event","event":"CHMOD         \"/tmp/997912853/ca.crt-0\""}
{"level":"info","ts":"2025-03-04T12:23:32.409-0500","caller":"run/run.go:54","msg":"received event","event":"CHMOD         \"/tmp/742621040/tls.key-0\""}
{"level":"info","ts":"2025-03-04T12:23:36.924-0500","caller":"server/server.go:130","msg":"Updating SDS config. sdsClient is test-client. Snapshot version is 16241649556325798095"}
{"level":"info","ts":"2025-03-04T12:23:36.928-0500","caller":"run/run.go:79","msg":"watcher started","sslKeyFile":"/tmp/742621040/tls.key-0","sshCertFile":"/tmp/742621040/tls.crt-0","sslCaFile":"/tmp/742621040/ca.crt-0"}
{"level":"info","ts":"2025-03-04T12:23:36.928-0500","caller":"run/run.go:54","msg":"received event","event":"REMOVE        \"/tmp/742621040/tls.key-0\""}
{"level":"info","ts":"2025-03-04T12:23:38.422-0500","caller":"server/server.go:130","msg":"Updating SDS config. sdsClient is test-client. Snapshot version is 16694243763138062080"}
{"level":"info","ts":"2025-03-04T12:23:38.423-0500","caller":"run/run.go:79","msg":"watcher started","sslKeyFile":"/tmp/997912853/tls.key-0","sshCertFile":"/tmp/997912853/tls.crt-0","sslCaFile":"/tmp/997912853/ca.crt-0"}
{"level":"warn","ts":"2025-03-04T12:23:38.423-0500","caller":"run/run.go:81","msg":"{error 26 0  no such file or directory}"}
{"level":"warn","ts":"2025-03-04T12:23:38.424-0500","caller":"run/run.go:84","msg":"{error 26 0  no such file or directory}"}
{"level":"warn","ts":"2025-03-04T12:23:38.424-0500","caller":"run/run.go:87","msg":"{error 26 0  no such file or directory}"}
{"level":"info","ts":"2025-03-04T12:23:41.457-0500","caller":"server/server.go:130","msg":"Updating SDS config. sdsClient is test-client. Snapshot version is 7644406922477208950"}
{"level":"info","ts":"2025-03-04T12:23:41.458-0500","caller":"run/run.go:79","msg":"watcher started","sslKeyFile":"/tmp/742621040/tls.key-0","sshCertFile":"/tmp/742621040/tls.crt-0","sslCaFile":"/tmp/742621040/ca.crt-0"}
{"level":"info","ts":"2025-03-04T12:23:41.458-0500","caller":"run/run.go:54","msg":"received event","event":"CHMOD         \"/tmp/742621040/tls.key-0\""}
{"level":"info","ts":"2025-03-04T12:23:42.454-0500","caller":"server/server.go:86","msg":"stopping sds server on 127.0.0.1:8236\n"}
•

Ran 3 of 3 Specs in 34.193 seconds
SUCCESS! -- 3 Passed | 0 Failed | 0 Pending | 0 Skipped
--- FAIL: TestSDS_E2E_Server (34.20s)
    testing.go:1490: race detected during execution of test
FAIL
FAIL	github.com/kgateway-dev/kgateway/v2/internal/sds/pkg/run	34.268s
FAIL
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@timflannagan