Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Retry io.EOF error #739

Merged
merged 1 commit into from
Apr 11, 2024
Merged

Retry io.EOF error #739

merged 1 commit into from
Apr 11, 2024

Conversation

yhabteab
Copy link
Member

Otherwise, Icinga DB will crash immediately!

2024-04-11T09:08:45.502+0200    INFO    history-sync    Synced 104 notification history items
2024-04-11T09:08:47.806+0200    WARN    database        Can't connect to database. Retrying     {"error": "pq: the database system is shutting down"}
2024-04-11T09:08:47.895+0200    WARN    database        Can't connect to database. Retrying     {"error": "read tcp 127.0.0.1:53259->127.0.0.1:5432: read: connection reset by peer"}
2024-04-11T09:08:48.095+0200    WARN    database        Can't connect to database. Retrying     {"error": "read tcp 127.0.0.1:53260->127.0.0.1:5432: read: connection reset by peer"}
2024-04-11T09:08:48.236+0200    FATAL   main    EOF
can't retry
github.com/icinga/icingadb/pkg/retry.WithBackoff
        /Users/yhabteab/Workspace/go/icingadb/pkg/retry/retry.go:81
github.com/icinga/icingadb/pkg/icingadb.RetryConnector.Connect
        /Users/yhabteab/Workspace/go/icingadb/pkg/icingadb/driver.go:42
database/sql.(*DB).conn
        /opt/homebrew/Cellar/go/1.22.1/libexec/src/database/sql/sql.go:1415
database/sql.(*DB).exec
        /opt/homebrew/Cellar/go/1.22.1/libexec/src/database/sql/sql.go:1679
database/sql.(*DB).ExecContext.func1
        /opt/homebrew/Cellar/go/1.22.1/libexec/src/database/sql/sql.go:1662
database/sql.(*DB).retry
        /opt/homebrew/Cellar/go/1.22.1/libexec/src/database/sql/sql.go:1573
database/sql.(*DB).ExecContext
        /opt/homebrew/Cellar/go/1.22.1/libexec/src/database/sql/sql.go:1661
github.com/jmoiron/sqlx.NamedExecContext
        /Users/yhabteab/Workspace/go/pkg/mod/github.com/jmoiron/[email protected]/named_context.go:131
github.com/jmoiron/sqlx.(*DB).NamedExecContext
        /Users/yhabteab/Workspace/go/pkg/mod/github.com/jmoiron/[email protected]/sqlx_context.go:134
github.com/icinga/icingadb/pkg/icingadb.(*DB).NamedBulkExec.func1.(*DB).NamedBulkExec.func1.1.2.1
        /Users/yhabteab/Workspace/go/icingadb/pkg/icingadb/db.go:398
github.com/icinga/icingadb/pkg/retry.WithBackoff
        /Users/yhabteab/Workspace/go/icingadb/pkg/retry/retry.go:59
github.com/icinga/icingadb/pkg/icingadb.(*DB).NamedBulkExec.func1.(*DB).NamedBulkExec.func1.1.2
        /Users/yhabteab/Workspace/go/icingadb/pkg/icingadb/db.go:395
golang.org/x/sync/errgroup.(*Group).Go.func1
        /Users/yhabteab/Workspace/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78
runtime.goexit
        /opt/homebrew/Cellar/go/1.22.1/libexec/src/runtime/asm_arm64.s:1222
can't connect to database
github.com/icinga/icingadb/pkg/icingadb.RetryConnector.Connect
        /Users/yhabteab/Workspace/go/icingadb/pkg/icingadb/driver.go:42
database/sql.(*DB).conn
        /opt/homebrew/Cellar/go/1.22.1/libexec/src/database/sql/sql.go:1415
database/sql.(*DB).exec
        /opt/homebrew/Cellar/go/1.22.1/libexec/src/database/sql/sql.go:1679
database/sql.(*DB).ExecContext.func1
        /opt/homebrew/Cellar/go/1.22.1/libexec/src/database/sql/sql.go:1662
database/sql.(*DB).retry
        /opt/homebrew/Cellar/go/1.22.1/libexec/src/database/sql/sql.go:1573
database/sql.(*DB).ExecContext
        /opt/homebrew/Cellar/go/1.22.1/libexec/src/database/sql/sql.go:1661
github.com/jmoiron/sqlx.NamedExecContext
        /Users/yhabteab/Workspace/go/pkg/mod/github.com/jmoiron/[email protected]/named_context.go:131
github.com/jmoiron/sqlx.(*DB).NamedExecContext
        /Users/yhabteab/Workspace/go/pkg/mod/github.com/jmoiron/[email protected]/sqlx_context.go:134
github.com/icinga/icingadb/pkg/icingadb.(*DB).NamedBulkExec.func1.(*DB).NamedBulkExec.func1.1.2.1
        /Users/yhabteab/Workspace/go/icingadb/pkg/icingadb/db.go:398
github.com/icinga/icingadb/pkg/retry.WithBackoff
        /Users/yhabteab/Workspace/go/icingadb/pkg/retry/retry.go:59
github.com/icinga/icingadb/pkg/icingadb.(*DB).NamedBulkExec.func1.(*DB).NamedBulkExec.func1.1.2
        /Users/yhabteab/Workspace/go/icingadb/pkg/icingadb/db.go:395
golang.org/x/sync/errgroup.(*Group).Go.func1
        /Users/yhabteab/Workspace/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78
runtime.goexit
        /opt/homebrew/Cellar/go/1.22.1/libexec/src/runtime/asm_arm64.s:1222
can't perform "INSERT INTO \"state_history\" (\"event_time\", \"previous_hard_state\", \"endpoint_id\", \"id\", \"service_id\", \"soft_state\", \"previous_soft_state\", \"scheduling_source\", \"host_id\", \"state_type\", \"max_check_attempts\", \"check_source\", \"object_type\", \"environment_id\", \"hard_state\", \"check_attempt\", \"output\", \"long_output\") VALUES (:event_time,:previous_hard_state,:endpoint_id,:id,:service_id,:soft_state,:previous_soft_state,:scheduling_source,:host_id,:state_type,:max_check_attempts,:check_source,:object_type,:environment_id,:hard_state,:check_attempt,:output,:long_output) ON CONFLICT ON CONSTRAINT pk_state_history DO UPDATE SET \"id\" = EXCLUDED.\"id\""
github.com/icinga/icingadb/internal.CantPerformQuery
        /Users/yhabteab/Workspace/go/icingadb/internal/internal.go:30
github.com/icinga/icingadb/pkg/icingadb.(*DB).NamedBulkExec.func1.(*DB).NamedBulkExec.func1.1.2.1
        /Users/yhabteab/Workspace/go/icingadb/pkg/icingadb/db.go:400
github.com/icinga/icingadb/pkg/retry.WithBackoff
        /Users/yhabteab/Workspace/go/icingadb/pkg/retry/retry.go:59
github.com/icinga/icingadb/pkg/icingadb.(*DB).NamedBulkExec.func1.(*DB).NamedBulkExec.func1.1.2
        /Users/yhabteab/Workspace/go/icingadb/pkg/icingadb/db.go:395
golang.org/x/sync/errgroup.(*Group).Go.func1
        /Users/yhabteab/Workspace/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78
runtime.goexit
        /opt/homebrew/Cellar/go/1.22.1/libexec/src/runtime/asm_arm64.s:1222
can't retry
github.com/icinga/icingadb/pkg/retry.WithBackoff
        /Users/yhabteab/Workspace/go/icingadb/pkg/retry/retry.go:81
github.com/icinga/icingadb/pkg/icingadb.(*DB).NamedBulkExec.func1.(*DB).NamedBulkExec.func1.1.2
        /Users/yhabteab/Workspace/go/icingadb/pkg/icingadb/db.go:395
golang.org/x/sync/errgroup.(*Group).Go.func1
        /Users/yhabteab/Workspace/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78
runtime.goexit
        /opt/homebrew/Cellar/go/1.22.1/libexec/src/runtime/asm_arm64.s:1222
exit status 1

refs #698

@yhabteab yhabteab added the bug Something isn't working label Apr 11, 2024
@yhabteab yhabteab requested a review from oxzi April 11, 2024 07:14
@cla-bot cla-bot bot added the cla/signed label Apr 11, 2024
@yhabteab yhabteab added this to the 1.2.0 milestone Apr 11, 2024
@yhabteab yhabteab requested a review from julianbrost April 11, 2024 07:15
@yhabteab yhabteab force-pushed the retry-io-eof-error branch from 7806d7f to 1b380bb Compare April 11, 2024 08:09
julianbrost
julianbrost approved these changes Apr 11, 2024
View reviewed changes
Copy link
Contributor

@julianbrost julianbrost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wasn't able to quickly reproduce this specific error but it sounds plausible.

How long is it until we just replace the whole function with return true?

@julianbrost julianbrost merged commit 919871f into main Apr 11, 2024
31 checks passed
@julianbrost julianbrost deleted the retry-io-eof-error branch April 11, 2024 08:52
@julianbrost julianbrost mentioned this pull request Apr 11, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@oxzi oxzi oxzi approved these changes

@julianbrost julianbrost julianbrost approved these changes

Assignees
No one assigned
Labels
bug Something isn't working cla/signed
Projects
None yet
Milestone
1.2.0
Development

Successfully merging this pull request may close these issues.
3 participants
@yhabteab @julianbrost @oxzi