Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clean up S3 Bucket - and other places #251

Open
johnml1135 opened this issue Jan 3, 2024 · 0 comments
Open

Clean up S3 Bucket - and other places #251

johnml1135 opened this issue Jan 3, 2024 · 0 comments

Comments

@johnml1135
Copy link
Collaborator

There should be an agent that periodically runs that clears data that is no longer used in the S3 bucket and in the main /var/lib folders. It should include:

  • Getting rid of files with no Mongo DB entry
  • Getting rid of engines with no Mongo DB entry in /var/lib/engines
  • Getting rid of S3 bucket experiments with no Mongo DB entry, or that have been completed successfully and finished.

Note that:

  • We need to ensure that the Mongo DB entry will be created before the files are created, otherwise, we could have race conditions and delete files before they are entered into the database. Same with pulling all files from the S3 bucket before calling the job "ready do delete" or some such thing.
  • This job should run periodically (once a day?) to clean up files. It needs to clear files even if there was an unexpected shutdown.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: 🆕 New
Development

No branches or pull requests

1 participant