Build Reconciler Errors while waiting on upload

[nstogner]

[manager] 2023-08-16T23:43:25Z  ERROR   Reconciler error        {"controller": "notebook", "controllerGroup": "substratus.ai", "controllerKind": "Notebook", "Notebook": {"name":"example","namespace":"default"}, "namespace": "default", "name": "example", "reconcileID": "98e13661-b56f-4b99-bdb4-16153b0b1d29", "error": "getting storage object md5: calling the sci service to GetObjectMd5: rpc error: code = Unknown desc = storage: object doesn't exist"}
[manager] sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
[manager]       /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
[manager] sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
[manager]       /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:265
[manager] sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
[manager]       /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:226

How to reproduce:

With controller stopped:

kubectl notebook -d .

Wait for it to attempt to upload... Ctrl-C

Start controller.

Proposed Solution:

SCI could return a Not Found gRPC status code and the controller could check for this error and avoid returning an error in this case (which would avoid sending the controller into retry loop).