Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Close bucket store for users with no blocks synced #5674

Closed

Conversation

harry671003
Copy link
Contributor

@harry671003 harry671003 commented Nov 23, 2023

What this PR does:

  • If the tenant_cleanup_delay is large (say 24h), a deleted tenant can be continued to be synced by the store-gateways.
  • This is because the s3://<bucket>/<tenantID>/markers/tenant-deletion-mark.json will be kept for 24h after the tenant has been deleted.
  • If there are a lots of users being created and deleted in the cluster (canary testing), there could be a lot of deleted users synced by the store-gateway.
  • One of the problems with having too many active users is the number of metrics that store-gateways will keep around. If there are 50K deleted users synced by a store-gateway and a total of 300 store-gateways in the cluster that'll be 15 million series.
  • This change ensures that if no blocks are loaded for a tenant, their bucket stores can be closed immediately and the blocks_loaded metrics can be removed.

Which issue(s) this PR fixes:
Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Signed-off-by: 🌲 Harry 🌊 John 🏔 <[email protected]>
Signed-off-by: 🌲 Harry 🌊 John 🏔 <[email protected]>
@harry671003
Copy link
Contributor Author

This is no longer required if the tenant-delete-mark.json can be moved outside the tenant dir.
#5676

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant