[MDB-27106] Use system.remote_data_paths table for cleaning S3
#94
Conversation
aalexfvk
force-pushed
the
MDB-27106_Use_remote_data_paths_system_table_for_cleaning_s3
branch
5 times, most recently
from
e5347d7 to
2211ce4
Compare
aalexfvk
force-pushed
the
MDB-27106_Use_remote_data_paths_system_table_for_cleaning_s3
branch
from
140bb8e to
9df88c1
Compare
aalexfvk
changed the title
remote_data_paths system table for cleaning S3system.remote_data_paths table for cleaning S3
| try: | ||
| execute_query( | ||
| ctx, | ||
| f"CREATE TABLE IF NOT EXISTS {listing_table} (obj_path String) ENGINE MergeTree ORDER BY obj_path", |
There was a problem hiding this comment.
In what database this table will be created? Suggest to make it configurable through ch-tools configuration file (https://github.com/yandex/ch-tools/blob/master/ch_tools/common/config.py).
There was a problem hiding this comment.
Yes, good point. Done it.
It makes sense when we have different listing tables for different clusters/hosts, but script command remains the same.
| default=False, | ||
| help="List objects that are not referenced in the metadata.", | ||
| ) | ||
| @object_storage_group.command("clean") |
There was a problem hiding this comment.
What is the reason for removing the list command? afaik, we have no alternative.
There was a problem hiding this comment.
This command is useless for now. Before it was intended for generating a list of orpahned objects to stdout for reading by another cleaning script. But now we do this in one clean command.
Nevertheless, we can reproduce this functionality for now by chadmin object-storage clean --keep-paths --dry-run.
| for _, obj in s3_object_storage_iterator( | ||
| ctx.obj["disk_configuration"], object_name_prefix=prefix | ||
| ): | ||
| if obj.last_modified > now - to_time: |
There was a problem hiding this comment.
In the knobs description:
"Objects with a modification time falling interval [now - from_time, now - to_time] are considered.
But here now-to_time is the lower bound.
There was a problem hiding this comment.
It is an upper bound of an interval
ch-tools/ch_tools/chadmin/cli/object_storage_group.py
Lines 238 to 239 in 3dfdae2
Actually we filter objects newer than now - 24h point in time and pass objects created more than 24h ago.
aalexfvk
force-pushed
the
MDB-27106_Use_remote_data_paths_system_table_for_cleaning_s3
branch
2 times, most recently
from
1c0d66b to
3dfdae2
Compare
aalexfvk
force-pushed
the
MDB-27106_Use_remote_data_paths_system_table_for_cleaning_s3
branch
from
3dfdae2 to
a486698
Compare
aalexfvk
force-pushed
the
MDB-27106_Use_remote_data_paths_system_table_for_cleaning_s3
branch
from
4474325 to
3798e77
Compare
aalexfvk
deleted the
MDB-27106_Use_remote_data_paths_system_table_for_cleaning_s3
branch
Use
system.remote_data_pathstable for enumerating objects known to ClickHouse.Enumerated objects from S3 is placed into table in ClickHouse. And selecting of objects presented only on S3 is performed by ANTIJOIN query with
system.remote_data_pathstable.TODO (not implemented in the PR):
requestsmight not throw an error while disconnecting while stream request)Note:
PR is big due to updating of
poetry.lock.