Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up cleanup-removed-hosts-metadata script #89

Merged
merged 5 commits into from
Jan 11, 2024

Conversation

MikhailBurdukov
Copy link
Contributor

Performance improvement for cleanup-removed-hosts-metadata. In this implementation we unite the nodes to delete in transactions to do single operation for batch of nodes.

Script for time comparison:
cat populate.sh

clickhouse client --query "DROP DATABASE IF EXISTS test_db"
clickhouse client --query "CREATE DATABASE test_db"

for i in $(seq 1 $1);
do
    
	
    clickhouse client --query "CREATE TABLE test_db.test$i (a int, b int) engine=ReplicatedMergeTree('/clickhouse/tables/{shard}/test_db.table$i', '{replica}') order by a partition by a"
    clickhouse client --query "SYSTEM STOP MERGES test_db.table$1"
    for j in $(seq 1 $2);
    do
      clickhouse client --query "INSERT INTO test_db.test1 SELECT rand()%10000, rand(1)%1000 FROM numbers(100)"
    done

done
sudo service clickhouse-server stop

Results:
cmd : time chadmin zookeeper cleanup-removed-hosts-metadata <host>

data new old
./populate.sh 10 20 0m15.114s 2m46.153s
./populate.sh 20 2 0m4.864s 0m51.093s
./populate.sh 2 40 0m8.673s 5m54.860s
./populate.sh 15 15 2m46.116s 8m42.093s

@MikhailBurdukov MikhailBurdukov marked this pull request as ready for review January 9, 2024 14:13
@aalexfvk aalexfvk merged commit baaf843 into main Jan 11, 2024
@aalexfvk aalexfvk deleted the MDB-11396-cleanup-removed-hosts-metadata-speedup branch January 11, 2024 17:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants