Deploy/use central secret storage

[pat-s]

E.g. https://github.com/Infisical/infisical

[anbraten]

We currently use an encrypted file to store secrets. This currently has the huge benefit for us that it does not need a server as our infrastructure is pretty minimal using a single server. If this server would also hosts tools to deploy itself, it could easily bring us into chicken-egg situations, I guess. The same thing somehow applies to #36.

[pat-s]

Only HA (partly) solves the chicken/egg problem. And event there, some initial state must be provisioned manually once.

In case of failures of the central deployment app or secret storage, you always have local ssh access or backups to bring this one back up. These are then of course essential and required to provision all the rest.

Backups is actually a topic on its own. The cheapest way is to rsync the persistent data to s3. The combination of the above provides a robust architecture which can be restored manually in case of unforeseen events.