Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

In kubernetes when killing pod with goffys container, pod stuck in terminating. exit code 137. #795

Open
dpointk opened this issue Apr 17, 2024 · 2 comments

Comments

@dpointk
Copy link

dpointk commented Apr 17, 2024

We have an issue with pods running goofys as a sidecar in kubernetes. When we delete the pod, it takes time or stuck shutting down the goofys container. This is due to goofys connection to s3 endpoint not being terminated properly.
We weren't able to reproduce it in hyperscalers (aws,gcp,etc) .

The normal behavior of goofys when sent a sigkill , is to exit with exit code 137.
We're trying to figure out if there's a way to avoid having this exit code in goofys so that our pods will not be stuck on terminating and be deleted properly.

goofys version 0.23.1
running command: goofys -f --dir-mode 0777 --file-mode 0777 -o allow_other --debug_s3 --endpoint https://xxx.xxx.xxx bucket_name /data

The pod is running as privileged in kubernetes.

We can rule out OOM issues as this was checked. It also happens with empty buckets and with new pods that were just spun up.

Any assistance will be appreciated.

@AsoTora
Copy link

AsoTora commented May 13, 2024

+1, I cant figure it out yet. Are you also running in the Alpine-based container @dpointk?

@kahing
Copy link
Owner

kahing commented Jul 24, 2024

a process cannot handle SIGKILL, so if k8s is sending goofys SIGKILL there's nothing it can do

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants