Sync server issue

[aKardasz]

We ran across an issue where all of a sudden we started receiving:

2024-10-18T15:57:58.845Z [INFO]  secrets.vault-plugin-secrets-nats.vault-plugin-secrets-nats_166cbb0b.vault-plugin-secrets-nats.vault-plugin-secrets-nats: Periodic: starting periodic func for syncing accounts to nats: timestamp=2024-10-18T15:57:58.844Z

2024-10-18T15:59:28.844Z [ERROR] rollback: error rolling back: path=nats-secrets/ error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"

This affected the syncing of all accounts, at first we thought it may be because we were trying to sync 103 accounts, so we deleted those and still had similar errors. So then I looked at the last account updated, and it was updated with a user with bad permissions config ( '$JS.STREAM.LIST' was returned as '.STREAM.LIST', and this is teh case for all the '$JS.' in that particular config) .

We believe this is what caused the error and once it failed it caused the rest of the sync loop to fail as well, however could not pinpoint exactly in the code were it occured, I will be working on a better example here later today.