Skip to content

Commit

Permalink
feat: enable watch retries on Omni side
Browse files Browse the repository at this point in the history 
Enable retry logger in `omnictl`, now if Omni goes down, `omnictl` can
correctly restart watches:

```
$ _out/omnictl-linux-amd64 get users -w
*   NAMESPACE   TYPE   ID                                     VERSION
+   default     User   be9d5331-74b9-4dc8-a4eb-387107faf262   1
+   default     User   cad2763b-1477-4bd5-9bc9-b3a369e39ffd   1
+   default     User   d4a0e14c-5b76-43ad-9b81-08a95cdd6731   1
2024-11-05T20:04:11.309+0400	WARN	client/client.go:504	watch retrying	{"error": "rpc error: code = Internal desc = server closed the stream without sending trailers", "bookmark": "AAAAAAAAASw=", "backoff": "678.945448ms", "namespace": "default", "type": "Users.omni.sidero.dev"}
2024-11-05T20:04:11.990+0400	WARN	client/client.go:504	watch retrying	{"error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 100.119.35.2:443: connect: connection refused\"", "bookmark": "AAAAAAAAASw=", "backoff": "979.468482ms", "namespace": "default", "type": "Users.omni.sidero.dev"}
2024-11-05T20:04:12.970+0400	WARN	client/client.go:504	watch retrying	{"error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 100.119.35.2:443: connect: connection refused\"", "bookmark": "AAAAAAAAASw=", "backoff": "850.913234ms", "namespace": "default", "type": "Users.omni.sidero.dev"}
2024-11-05T20:04:13.821+0400	WARN	client/client.go:504	watch retrying	{"error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 100.119.35.2:443: connect: connection refused\"", "bookmark": "AAAAAAAAASw=", "backoff": "1.683608302s", "namespace": "default", "type": "Users.omni.sidero.dev"}
2024-11-05T20:04:15.505+0400	WARN	client/client.go:504	watch retrying	{"error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 100.119.35.2:443: connect: connection refused\"", "bookmark": "AAAAAAAAASw=", "backoff": "3.759619924s", "namespace": "default", "type": "Users.omni.sidero.dev"}
2024-11-05T20:04:19.266+0400	WARN	client/client.go:504	watch retrying	{"error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 100.119.35.2:443: connect: connection refused\"", "bookmark": "AAAAAAAAASw=", "backoff": "3.985097297s", "namespace": "default", "type": "Users.omni.sidero.dev"}
2024-11-05T20:04:23.253+0400	WARN	client/client.go:504	watch retrying	{"error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 100.119.35.2:443: connect: connection refused\"", "bookmark": "AAAAAAAAASw=", "backoff": "4.74055025s", "namespace": "default", "type": "Users.omni.sidero.dev"}
2024-11-05T20:04:27.995+0400	WARN	client/client.go:504	watch retrying	{"error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 100.119.35.2:443: connect: connection refused\"", "bookmark": "AAAAAAAAASw=", "backoff": "7.308402811s", "namespace": "default", "type": "Users.omni.sidero.dev"}
2024-11-05T20:04:35.309+0400	WARN	client/client.go:504	watch retrying	{"error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 100.119.35.2:443: connect: connection refused\"", "bookmark": "AAAAAAAAASw=", "backoff": "8.806341922s", "namespace": "default", "type": "Users.omni.sidero.dev"}
    default   User   cad2763b-1477-4bd5-9bc9-b3a369e39ffd   2
-   default   User   cad2763b-1477-4bd5-9bc9-b3a369e39ffd   2
```

Signed-off-by: Andrey Smirnov <[email protected]>
  • Loading branch information
@smira
smira committed Nov 5, 2024
1 parent 23ccdb5 commit 3e8bc8d
Show file tree
Hide file tree
Showing 7 changed files with 34 additions and 10 deletions.
2 changes: 1 addition & 1 deletion client/go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ require (
github.com/ProtonMail/gopenpgp/v2 v2.7.5
github.com/adrg/xdg v0.5.3
github.com/blang/semver v3.5.1+incompatible
github.com/cosi-project/runtime v0.6.4
github.com/cosi-project/runtime v0.7.1
github.com/dustin/go-humanize v1.0.1
github.com/fatih/color v1.18.0
github.com/gertd/go-pluralize v0.2.1
Expand Down
4 changes: 2 additions & 2 deletions client/go.sum
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,8 @@ github.com/containerd/go-cni v1.1.10 h1:c2U73nld7spSWfiJwSh/8W9DK+/qQwYM2rngIhCy
github.com/containerd/go-cni v1.1.10/go.mod h1:/Y/sL8yqYQn1ZG1om1OncJB1W4zN3YmjfP/ShCzG/OY=
github.com/containernetworking/cni v1.2.3 h1:hhOcjNVUQTnzdRJ6alC5XF+wd9mfGIUaj8FuJbEslXM=
github.com/containernetworking/cni v1.2.3/go.mod h1:DuLgF+aPd3DzcTQTtp/Nvl1Kim23oFKdm2okJzBQA5M=
github.com/cosi-project/runtime v0.6.4 h1:roifc5e+Q1+72EI36BYSRT9aXyskU+coiKHeoBBWkMg=
github.com/cosi-project/runtime v0.6.4/go.mod h1:EMLs8a55tJ6zA4UyDbRsTvXBd6UIlNwZfCVGvCyiXK8=
github.com/cosi-project/runtime v0.7.1 h1:cOF2/ljLa0NPQV6/S1RZHcSUXM0UyOlD5F3fwwYSQEg=
github.com/cosi-project/runtime v0.7.1/go.mod h1:EMLs8a55tJ6zA4UyDbRsTvXBd6UIlNwZfCVGvCyiXK8=
github.com/cpuguy83/go-md2man/v2 v2.0.4/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
github.com/cpuguy83/go-md2man/v2 v2.0.5 h1:ZtcqGrnekaHpVLArFSe4HK5DoKx1T0rq2DwVB0alcyc=
github.com/cpuguy83/go-md2man/v2 v2.0.5/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
Expand Down
14 changes: 13 additions & 1 deletion client/pkg/client/omni/omni.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import (
"github.com/cosi-project/runtime/api/v1alpha1"
"github.com/cosi-project/runtime/pkg/state"
"github.com/cosi-project/runtime/pkg/state/protobuf/client"
"go.uber.org/zap"
"google.golang.org/grpc"
"google.golang.org/grpc/metadata"

Expand All @@ -27,6 +28,7 @@ import (

// Options defines additional Omni client options.
type Options struct {
retryLogger *zap.Logger
infraProviderID string
}

Expand All @@ -40,6 +42,13 @@ func WithProviderID(id string) Option {
}
}

// WithRetryLogger sets logger for retry operations.
func WithRetryLogger(lg *zap.Logger) Option {
return func(o *Options) {
o.retryLogger = lg
}
}

// Client for Omni resource API (COSI).
type Client struct {
conn *grpc.ClientConn
Expand All @@ -51,13 +60,16 @@ type Client struct {
func NewClient(conn *grpc.ClientConn, options ...Option) *Client {
c := &Client{
conn: conn,
options: Options{
retryLogger: zap.NewNop(),
},
}

for _, o := range options {
o(&c.options)
}

c.state = state.WrapCore(client.NewAdapter(v1alpha1.NewStateClient(c)))
c.state = state.WrapCore(client.NewAdapter(v1alpha1.NewStateClient(c), client.WithRetryLogger(c.options.retryLogger)))

return c
}
Expand Down
12 changes: 12 additions & 0 deletions client/pkg/omnictl/internal/access/client.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,10 @@ import (
"github.com/cosi-project/runtime/pkg/safe"
"github.com/cosi-project/runtime/pkg/state"
"github.com/siderolabs/go-api-signature/pkg/serviceaccount"
"go.uber.org/zap"

"github.com/siderolabs/omni/client/pkg/client"
"github.com/siderolabs/omni/client/pkg/client/omni"
"github.com/siderolabs/omni/client/pkg/omni/resources"
"github.com/siderolabs/omni/client/pkg/omni/resources/system"
"github.com/siderolabs/omni/client/pkg/omnictl/config"
Expand Down Expand Up @@ -112,6 +114,16 @@ func WithClient(f func(ctx context.Context, client *client.Client) error, client
url = endpointEnv
}

loggerCfg := zap.NewDevelopmentConfig()
loggerCfg.Development = false

logger, err := loggerCfg.Build()
if err != nil {
return err
}

opts = append(opts, client.WithOmniClientOptions(omni.WithRetryLogger(logger)))

client, err := client.New(url, opts...)
if err != nil {
return err
Expand Down
Binary file removed frontend/bun.lockb
View file
Binary file not shown.
4 changes: 2 additions & 2 deletions go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@ require (
github.com/blang/semver v3.5.1+incompatible
github.com/blang/semver/v4 v4.0.0
github.com/containers/image/v5 v5.32.2
github.com/cosi-project/runtime v0.6.4
github.com/cosi-project/state-etcd v0.3.2
github.com/cosi-project/runtime v0.7.1
github.com/cosi-project/state-etcd v0.4.0
github.com/crewjam/saml v0.4.14
github.com/dustin/go-humanize v1.0.1
github.com/emicklei/dot v1.6.2
Expand Down
8 changes: 4 additions & 4 deletions go.sum
View file
Original file line number Diff line number Diff line change
Expand Up @@ -109,10 +109,10 @@ github.com/coreos/go-semver v0.3.1 h1:yi21YpKnrx1gt5R+la8n5WgS0kCrsPp33dmEyHReZr
github.com/coreos/go-semver v0.3.1/go.mod h1:irMmmIw/7yzSRPWryHsK7EYSg09caPQL03VsM8rvUec=
github.com/coreos/go-systemd/v22 v22.5.0 h1:RrqgGjYQKalulkV8NGVIfkXQf6YYmOyiJKk8iXXhfZs=
github.com/coreos/go-systemd/v22 v22.5.0/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc=
github.com/cosi-project/runtime v0.6.4 h1:roifc5e+Q1+72EI36BYSRT9aXyskU+coiKHeoBBWkMg=
github.com/cosi-project/runtime v0.6.4/go.mod h1:EMLs8a55tJ6zA4UyDbRsTvXBd6UIlNwZfCVGvCyiXK8=
github.com/cosi-project/state-etcd v0.3.2 h1:8Lnj5dPIUzkv5DYFpKvkP+vX5O9M9sWthXQfoKJrF78=
github.com/cosi-project/state-etcd v0.3.2/go.mod h1:wHp8V3OvGRILYvbJEmhMt0AFuByaGNkxXLDc61/+yFM=
github.com/cosi-project/runtime v0.7.1 h1:cOF2/ljLa0NPQV6/S1RZHcSUXM0UyOlD5F3fwwYSQEg=
github.com/cosi-project/runtime v0.7.1/go.mod h1:EMLs8a55tJ6zA4UyDbRsTvXBd6UIlNwZfCVGvCyiXK8=
github.com/cosi-project/state-etcd v0.4.0 h1:/zf2mbqVHnp32wWDBaFGa5U1+ZwyenBzeZOgrxmNzhY=
github.com/cosi-project/state-etcd v0.4.0/go.mod h1:V3UiPxXs2Nl3kls8TSnk7ABKx68fCjNZDTu2Ux9CGK8=
github.com/cpuguy83/go-md2man/v2 v2.0.4/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
github.com/cpuguy83/go-md2man/v2 v2.0.5 h1:ZtcqGrnekaHpVLArFSe4HK5DoKx1T0rq2DwVB0alcyc=
github.com/cpuguy83/go-md2man/v2 v2.0.5/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
Expand Down

0 comments on commit 3e8bc8d

Please sign in to comment.