Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

compactor_blocks_retention_period not deleting any data #6295

Open
adriendinzey opened this issue Oct 30, 2024 · 3 comments
Open

compactor_blocks_retention_period not deleting any data #6295

adriendinzey opened this issue Oct 30, 2024 · 3 comments

Comments

@adriendinzey
Copy link

adriendinzey commented Oct 30, 2024

Describe the bug
After setting this configuration to 1d and then monitoring it for days after, none of the data is being deleted from the s3 bucket. In the logs there is no mention of "retention" or even "deletion"
To Reproduce
Steps to reproduce the behavior:

  1. Set compactor_blocks_retention_period to 1d
  2. Wait a day or more

Expected behavior
I would expect the data older than a day would bd deleted and I would see in the logs that the retention period is being applied.

Environment:
Version: cortex-1.18.1_amd64

Additional Context
In the config file I have
limits:
compactor_blocks_retention_period: 1d

@yeya24
Copy link
Contributor

yeya24 commented Nov 1, 2024

Hey @adriendinzey, this is not expected. How do you run your compactor? You have to run your compactor in a dedicated service instead of the single binary mode.

Does your compactor have any error?

@adriendinzey
Copy link
Author

The only error I saw in the logs was a failed health check, not sure if that is related. I was previously having issues getting cortex to properly read the compactor_blocks_retention_period, and those errors have since been resolved. Due to that, I believe this indicates the configuration for that flag is being parsed and accepted properly, but the deletion is not occuring.

@yeya24
Copy link
Contributor

yeya24 commented Nov 24, 2024

Did you see any log like below?

applying retention
applied retention: marking block for deletion

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants