This repository has been archived by the owner on Mar 3, 2023. It is now read-only.
Duplicate files installed bumping up install size by ~190M #3670
Comments
|
Is the script accounting for symlinks? I believe some of the contents in |
Good question, but not in these cases. Here's all the symlinks that exist in case it helps a clean up: # made during heron-install.sh
ln -s /usr/local/heron/bin/heron{,-apiserver,-explorer,-tracker,-ui} /usr/local/bin/
# made in the Dockerfile
ln -s /usr/local/heron/dist/heron-core /heron/
ln -s /usr/local/heron/{examples,release.yaml} /heron/
ln -s /usr/local/heron/{bin,conf,dist,lib,release.yaml /heron/heron-tools/ |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
The current install is ~816M as seen in the docker images, with ~20% duplicate files. It may be worth cleaning up the install (preferably by reworking the install script) if anyone wants to reduce container sizes:
( cd /usr/local; join -j1 -t' ' \ <(find . -type f | xargs md5sum --binary | cut -d' ' -f1 | sort | uniq --repeated) \ <(find . -type f | xargs md5sum --binary | sort) \ | awk 'last != $1 { printf ": " system( "echo " size * (count - 1) " | numfmt --to=iec" ); cmd="stat -c %s " substr($2, 2); cmd | getline size; last=$1; count=0; } { count++; printf "%s ", substr($2, 2); } END { printf "total duplicated size: " system( "echo " size * (count - 1) " | numfmt --to=iec" ); } ' ) | sort -t: -h -k2Gives a total of 190M duplicated:
most of which (182.9M) is between
/usr/local/heron/lib/and/usr/local/heron/dist/heron-core/lib/$ ( for dir in /usr/local/heron/lib/ /usr/local/heron/dist/heron-core/lib/; do (cd "$dir" && find . -type f -exec sh -c 'ls -sh {} | tail -n1' \; ) done ) | sort | uniq --count --repeated 2 11M ./statemgr/heron-zookeeper-statemgr.jar 2 13M ./metricscachemgr/heron-metricscachemgr.jar 2 14M ./scheduler/heron-mesos-scheduler.jar 2 15M ./scheduler/heron-nomad-scheduler.jar 2 36M ./scheduler/heron-kubernetes-scheduler.jar 2 45M ./downloaders/heron-downloader.jar 2 5.7M ./packing/heron-binpacking-packing.jar 2 5.7M ./packing/heron-roundrobin-packing.jar 2 5.7M ./statemgr/heron-localfs-statemgr.jar 2 7.8M ./scheduler/heron-scheduler.jar 2 8.0M ./scheduler/heron-local-scheduler.jar 2 8.0M ./scheduler/heron-marathon-scheduler.jar 2 8.0M ./scheduler/heron-slurm-scheduler.jarThe text was updated successfully, but these errors were encountered: