Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Delete objects by path from s3 object storage #86

Closed
wants to merge 1 commit into from

Conversation

MedvedewEM
Copy link
Contributor

Sometimes we need to delete data from S3 Object Storage manually when replicas with local metadata already deleted and no backups refer to data.
So there is no another way to clear S3 except explicitly iterate all objects by prefix and delete it.

Now it works like:

 # chadmin object-storage list
{"object": {"key": "cloud_storage/e4u21vnmivo70tppvttj/shard1/ees/ewcpleokxjcsldqkgtiqpxbvpwlsp", "size": 1, "last_modified": "2023-12-12 16:49:00.117000+00:00"}, "files": ["/var/lib/clickhouse/disks/object_storage/store/9a0/9a0182da-9530-44f5-9c70-6dd595e31f9d/202312_2_2_0/metadata_version.txt"]}
{"object": {"key": "cloud_storage/e4u21vnmivo70tppvttj/shard1/jeu/patimrwqaqnaojatfzjmzgesojlur", "size": 100, "last_modified": "2023-12-12 16:49:00.113000+00:00"}, "files": ["/var/lib/clickhouse/disks/object_storage/store/9a0/9a0182da-9530-44f5-9c70-6dd595e31f9d/202312_2_2_0/columns.txt"]}
{"object": {"key": "cloud_storage/e4u21vnmivo70tppvttj/shard1/lzh/sjydftjppnoghiqdmelftexlubswr", "size": 4, "last_modified": "2023-12-12 16:49:00.120000+00:00"}, "files": ["/var/lib/clickhouse/disks/object_storage/store/9a0/9a0182da-9530-44f5-9c70-6dd595e31f9d/202312_2_2_0/minmax_pickup_date.idx"]}
{"object": {"key": "cloud_storage/e4u21vnmivo70tppvttj/shard1/tuy/rootgostdyrqdkcmgshbrduzpdzfn", "size": 88, "last_modified": "2023-12-12 16:49:00.112000+00:00"}, "files": ["/var/lib/clickhouse/disks/object_storage/store/9a0/9a0182da-9530-44f5-9c70-6dd595e31f9d/202312_2_2_0/data.bin"]}
{"object": {"key": "cloud_storage/e4u21vnmivo70tppvttj/shard1/tvl/oisrcewacvpvkwhrqvvyhhitbqxav", "size": 62, "last_modified": "2023-12-12 16:49:00.078000+00:00"}, "files": ["/var/lib/clickhouse/disks/object_storage/store/9a0/9a0182da-9530-44f5-9c70-6dd595e31f9d/202312_2_2_0/data.cmrk3"]}
{"object": {"key": "cloud_storage/e4u21vnmivo70tppvttj/shard1/vms/ljpfduakpcwyvbzcihluncyvcvkzn", "size": 4, "last_modified": "2023-12-12 16:49:00.078000+00:00"}, "files": ["/var/lib/clickhouse/disks/object_storage/store/9a0/9a0182da-9530-44f5-9c70-6dd595e31f9d/202312_2_2_0/partition.dat"]}
{"object": {"key": "cloud_storage/e4u21vnmivo70tppvttj/shard1/vwa/qprmeaqtxhskarsmupbnhpfbkwgmz", "size": 10, "last_modified": "2023-12-12 16:49:00.078000+00:00"}, "files": ["/var/lib/clickhouse/disks/object_storage/store/9a0/9a0182da-9530-44f5-9c70-6dd595e31f9d/202312_2_2_0/default_compression_codec.txt"]}
{"object": {"key": "cloud_storage/e4u21vnmivo70tppvttj/shard1/wmw/upddckttflhqvcdictycubzdutdlo", "size": 331, "last_modified": "2023-12-12 16:49:00.083000+00:00"}, "files": ["/var/lib/clickhouse/disks/object_storage/store/9a0/9a0182da-9530-44f5-9c70-6dd595e31f9d/202312_2_2_0/checksums.txt"]}
{"object": {"key": "cloud_storage/e4u21vnmivo70tppvttj/shard1/wpg/xujlxchstrwnmnhzfnwemtsceuety", "size": 235, "last_modified": "2023-12-12 16:49:00.108000+00:00"}, "files": ["/var/lib/clickhouse/disks/object_storage/store/9a0/9a0182da-9530-44f5-9c70-6dd595e31f9d/202312_2_2_0/serialization.json"]}
{"object": {"key": "cloud_storage/e4u21vnmivo70tppvttj/shard1/xbo/dsiqwlsxghdlsxbuahcnnblvwsksc", "size": 42, "last_modified": "2023-12-12 16:49:00.119000+00:00"}, "files": ["/var/lib/clickhouse/disks/object_storage/store/9a0/9a0182da-9530-44f5-9c70-6dd595e31f9d/202312_2_2_0/primary.cidx"]}
{"object": {"key": "cloud_storage/e4u21vnmivo70tppvttj/shard1/xfg/impnnnrxtqkskgkweaoarvlvadpsn", "size": 1, "last_modified": "2023-12-12 16:49:00.110000+00:00"}, "files": ["/var/lib/clickhouse/disks/object_storage/store/9a0/9a0182da-9530-44f5-9c70-6dd595e31f9d/202312_2_2_0/count.txt"]}

# chadmin object-storage clean --prefix=cloud_storage/e4u21vnmivo70tppvttj/shard1
Deleted 11 objects from bucket [cloud-storage-e4u21vnmivo70tppvttj]

PS fix util's method chunked, because current code does not work (infinite loop).

@MedvedewEM MedvedewEM self-assigned this Dec 12, 2023
@MedvedewEM MedvedewEM force-pushed the MDB-26347/object_storage_delete_path branch from 06c0fc9 to cf4fcdd Compare December 12, 2023 18:08
@MedvedewEM MedvedewEM marked this pull request as draft December 12, 2023 18:11
@MedvedewEM MedvedewEM force-pushed the MDB-26347/object_storage_delete_path branch 2 times, most recently from 0028675 to 86e3d86 Compare December 12, 2023 18:16
@MedvedewEM MedvedewEM force-pushed the MDB-26347/object_storage_delete_path branch from 86e3d86 to 7fe8b66 Compare December 12, 2023 18:20
@MedvedewEM MedvedewEM closed this Dec 12, 2023
@MedvedewEM MedvedewEM deleted the MDB-26347/object_storage_delete_path branch December 12, 2023 18:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant