-
Notifications
You must be signed in to change notification settings - Fork 30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Timeout functionality #51
Comments
IMHO it's a really interesting feature, other opinions? |
we might be able to simplify by using -numBytes, err := io.Copy(dst, object)
+err := Copy(ctx, dst, object) I am not sure if the code in the referenced blog post is any good. But I like the approach of abstracting this to a separate function that resembles |
@bradrydzewski I agree that would be a much simpler implementation and I'm happy to try and submit something to that nature. My only concern is, what if the plugin "hangs" on a different part of the function? We'd have to be very clear on the implementation that the timeout flag for the plugin was only a timeout for how long to wait for the copy function. To be specific, what if the minio drone-s3-cache/storage/s3/s3.go Lines 71 to 102 in 327bde5
Sure, we could swap to using the This could have unintended consequences if both the Do you have thoughts on that? |
I seem to be having a similar issue, when the AWS IAM role available to drone does not let him create a bucket : The logs indicate that Logs (seems to be hanging there)
Pipeline
|
Our IAM role is enabled - other s3 actions are being performaned from the same kubernetes cluster that our drone install runs in - and I think we are experiancing the same behavior as @Baccata describes.
|
update - this was solved by entering the endpoint for our region of aws. |
Currently, if we experience issues with our S3 cache service provider, it can cause the plugin to "hang". This causes the build to be stuck in a "running" state until the build reaches the timeout (usually 1 hour).
A single occurrence of the issue is not a problem, but when it happens across the entire enterprise, it causes the number of pending builds in the queue to grow massively because our agent capacity is consumed by all the "stuck" cache builds.
The logs produced in the step are:
This leads me to believe it's getting hung on line #93 because we never see the log message from line #99.
drone-s3-cache/storage/s3/s3.go
Lines 91 to 101 in 327bde5
To remediate this issue, we decided to create our own timeout feature that we embedded into the plugin and drone/drone-cache-lib.
Would this be a feature the Drone admins were willing to accept into the plugin? I've attached the changesets below so you can see how we went about resolving this issue.
Repo Changes:
drone/drone-cache-lib@master...jbrockopp:master
master...jbrockopp:master
The text was updated successfully, but these errors were encountered: