icingadb.DB: Retry Schema Checks

[oxzi]

During testing, I just encountered a race condition where my Galera cluster was not yet ready, causing the initial schema check to fail.

2024-04-11T08:13:40.401Z        INFO    icingadb        Starting Icinga DB daemon (1.1.1)
2024-04-11T08:13:40.401Z        INFO    icingadb        Connecting to database at 'mysql:3306'
2024-04-11T08:13:40.404Z        FATAL   icingadb        Error 1047 (08S01): WSREP has not yet prepared node for application use
can't check database schema version
github.com/icinga/icingadb/pkg/icingadb.(*DB).CheckSchema
        /go/src/github.com/Icinga/icingadb/pkg/icingadb/db.go:115
main.run
        /go/src/github.com/Icinga/icingadb/cmd/icingadb/main.go:74
main.main
        /go/src/github.com/Icinga/icingadb/cmd/icingadb/main.go:37
runtime.main
        /usr/local/go/src/runtime/proc.go:271
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1695
exit status 1

This change now also retries the initial cluster check.

References #698.

[lippserd]

We could also remove db#shouldRetry() as retry#Retryable() also handles driver.ErrBadConn.

[julianbrost]

We could also remove db#shouldRetry() as retry#Retryable() also handles driver.ErrBadConn.

Is that combined with the thumbs up a sign that you want to do it now?

[lippserd]

We could also remove db#shouldRetry() as retry#Retryable() also handles driver.ErrBadConn.

Is that combined with the thumbs up a sign that you want to do it now?

I would do that in a separate PR. @oxzi Can you take care of that please?

[oxzi]

@julianbrost: The 👍 was just a general sign of "yeah, that's something that somebody should sometimes do". It seems like the burden fall onto me.