429 Too Many Requests Rate limit exceeded, retry in 6 seconds

[victorgetz]

Since yesterday we have some issues.

Existing configurations now show "Got error from UptimeRobot: null"
In Debug Mode we see this: "429 Too Many Requests Rate limit exceeded, retry in 6 seconds"

My Guess is that Uptimerobot included a rate limit.

Any Idea how to solve that? Make the intervall smaller and add retry?

[MujahidRashid]

@victorgetz
i am getting the same error as well, how do you enable debug mode ?
i tried with export TF_LOG=trace, but i do not see the "too many request" error

[dlecan]

Same error for me!
I think this is an issue with the Uptimerobot API

Got response: &http.Response{Status:"429 Too Many Requests",
    StatusCode:429,
    Proto:"HTTP/2.0",
    ProtoMajor:2,
    ProtoMinor:0,
    Header:http.Header{"Cf-Cache-Status":[]string{"DYNAMIC"},
    "Cf-Ray":[]string{"676ff3b58e170100-AMS"},
    "Content-Security-Policy":[]string{"default-src 'self' http: https: data: blob: 'unsafe-inline'"},
    "Content-Type":[]string{"text/html; charset=utf-8"},
    "Date":[]string{"Fri,
    30 Jul 2021 16:27:01 GMT"},
    "Expect-Ct":[]string{"max-age=604800,
    report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\""},
    "Nel":[]string{"{\"report_to\":\"cf-nel\",\"max_age\":604800}"},
    "Referrer-Policy":[]string{"no-referrer-when-downgrade"},
    "Report-To":[]string{"{\"endpoints\":[{\"url\":\"https:\\/\\/a.nel.cloudflare.com\\/report\\/v3?s=...\"}],\"group\":\"cf-nel\",\"max_age\":604800}"},
    "Retry-After":[]string{"9"},
    "Server":[]string{"cloudflare"},
    "Strict-Transport-Security":[]string{"max-age=31536000; includeSubDomains; preload"},
    "Vary":[]string{"Accept-Encoding"},
    "X-Content-Type-Options":[]string{"nosniff"},
    "X-Frame-Options":[]string{"SAMEORIGIN"},
    "X-Ratelimit-Limit":[]string{"50"},
    "X-Ratelimit-Remaining":[]string{"-1"},
    "X-Ratelimit-Reset":[]string{"1627662431"},
    "X-Xss-Protection":[]string{"1; mode=block"}},
    Body:http.http2transportResponseBody{cs:(*http.http2clientStream)(0xc000304f20)},
    ContentLength:-1,
    TransferEncoding:[]string(nil),
    Close:false,
    Uncompressed:false,
    Trailer:http.Header(nil),
    Request:(*http.Request)(0xc000540b00),
    TLS:(*tls.ConnectionState)(0xc0000a1130)}
2021-07-30T18:27:01.828+0200 [DEBUG] provider.terraform-provider-uptimerobot: 2021/07/30 18:27:01 [DEBUG] Got body: "Rate limit exceeded,
retry in 9 seconds"

I will contact Uptimerobot support

[dlecan]

Uptimerobot answer 😭
Uptimerobot becomes useless if the quotas are not increased

[dlecan]

And the issue is exaggerated by #85, unreleased ...

[invidian]

I think it shouldn't be too hard to implement a retry on 429 for this provider, though 10 requests per minute seems like a serious limitation, which will increase Terraform execution time a lot... Maybe some client-side magic could be done to improve that though.

[dlecan]

Some hope

[clovnrian]

Hello guys, this is Mirek from UptimeRobot. We just increased the API rate limits for PRO users twice, which should be enough to use this plugin again.

Rate limits for FREE users are still the same (10 req/m)

[invidian]

It seems with combination with https://github.com/stakater/IngressMonitorController, iteracting with UptimeRobot API is now barely usable 😞

[invidian]

Not very production quality but I wrote a patch, which makes provider very slow when you're close to API limit, but usable: https://github.com/louy/terraform-provider-uptimerobot/compare/master...invidian:invidian/retry-on-rate-limit?expand=1

[dlecan]

@clovnrian it's an improvement, thank you.
But not enough to avoid errors during updates

Quotas could be different between GET and POST/PUT? (higher with updates)

@invidian interesting, have you opened a PR? I can't find one to comment

[clovnrian]

Our API allows only POST requests and we are not making difference between "GET" type of requests and requests which make some insert/update.

[kokarn]

This issue is a huge roadblock for us I'm afraid. Because of the sheer number of requests being run for us during any sort of terraform plan stage just to check the current status it quickly becomes unworkable.

Say we have 20-30 sites we monitor.
If 3 people are working, we have to be on the 100 plan just to run terraform plan

If we employ 2 more people we need to upgrade without adding any more monitors etc.
The same exponential growth can be seen if adding 20-30 more sites.

Is there any plans to allow us to purchase more API requests just for checks or adding other plans with increased API usage?

Another possible solution would perhaps be to just do the API request once, but not sure if that's possible with terraform providers

[adrianchifor]

Experiencing this as well for 15 monitors, the 10 req/min makes automation completely impossible @clovnrian

[clovnrian]

OK guys, I can see it's real issue for you, so let find some solution to make most of you happy.

I am not familiar with Terraform and this plugin, so can somebody please explain me how it's working? Ideally with some real world use-cases. I can assure you we don't want to sabotage this plugin in any way and we will try find solution for you.

@kokarn as I wrote above I am not familiar with Terraform, but from the little I know, it's tool for managing infrastructure with code. And it don't make sense to me why every working person would call the terraform plan. Can you please explain your use-case?

[kokarn]

@kokarn as I wrote above I am not familiar with Terraform, but from the little I know, it's tool for managing infrastructure with code. And it don't make sense to me why every working person would call the terraform plan. Can you please explain your use-case?

You are correct, it's a tool that manages infrastructure.

In larger teams & setups infrastructure might change in many different ways, so any time you propose a change you validate that towards the infrastructure in general.

That includes validation that the "state" of the infrastructure you have locally matches the live infrastructure.

This includes in our case 1000's of different pieces, everything from access rights for accounts in the cloud to integration with third parties and everything in between.

A very common use case for us is to add a new site running a pre-existing configuration.
Then I would that service to the template and run terraform plan to see what changes would be done in the whole infrastructure to accommodate my addition.
I get instant feedback telling me exactly what changes would be done.
I then validate those changes and either approve them and commit them or alter them and run terraform plan again.
Maybe I get an error that one cluster is out of resources, or maybe there's a certificate missing somewhere. It all depends on what others are doing at the same time.
This is just day2day work and is impacted directly by the API limits.

A much worse scenario is me and a colleague working on different services in the same infrastructure, but it's a new piece of code. Then there are many many more runs of terraform plan to validate all parts of it and adding all components needed to get it up and running. In such a scenario it's not uncommon to plan 2-3 times a minute.

I think a decent mental model is linting but for infrastructure. That's a bit of what plan does in this scenario. Getting rate-limited on linting is really annoying :D

[clovnrian]

Thanks for explanation @kokarn.

This use case make sense and I can imagine other users have similar one. Also I totally get your frustration, and I am sorry. From what I read in Terraform documentation, this is something which is probably impossible or very hard to fix in plugin code (if you don't want to add some delay after every request).

We will definitely look in to it and try to find solution where both sides will be happy.

[clovnrian]

Hello guys, we are still debating about the API rate limit, but after 2 minutes googling I found article about rate limiting in HTTP client.

I am not Golang developer, but at first look I think it can be easily integrated to this library. Yes, it will probably increase the execution time of terraform scrips, but it shouldn't be drastic.