HTTP 400 error when sending lots of requests at once.

[davidseverwright]

I've been having errors when sending multiple requests at once. It's a mix of HTTP 400 and timeouts.

In my setup, I have a queue runner sending reports. Every hour docker starts the export server and my queue runner, and then stops them both once the queue is empty.

I had the export server set to run 4 workers, and my queue runner was running 4 threads; there's no rate limiting so the 4 workers were getting hit non-stop.
With the old export server this was fine, with 3.0.0-beta.2 it works for about ~125 requests then became 99% errors.

I've now changed it to only have 1 worker and 1 queue runner, and it's now running without issue.
I've tried increasing the CPU and RAM allocated, seems to reduce the timeouts but still get the 400's.

All testing was done with around 600 charts being created
4 workers, 4 queues = bad (~125 queries, then very high errors.)
4 workers, 1 queue = fine
1 worker & 2 queues = eventually a few errors (~400 queries, then ~50% error for a while, then fine again)
1 worker & 1 queue = fine

The number of workers doesn't seem to matter, just the number of requests coming at once.

The attached log is with 4 threads sending requests, and --workers 4. 2vCPU and 8 GB of RAM, on AWS Fargate.
With just one core and 2GB it's about the same error rate but more timeouts. The timeouts also increase over time, so maybe related to the memory leak bug?

highcharts-error.log

[jszuminski]

Thanks for reporting!

Seems like it's related to: #412

[jszuminski]

@davidseverwright could you please share the following details:

1. Are you injecting resources (JS/CSS)?
2. Does the memory use increase over time for you?

[davidseverwright]

I'm not injecting any CSS/JS, and the memory usage is fluctuating a bit but isn't just climbing like the graph in #412.

With 4 workers and 4 jobs I get lots of errors and this:

With 4 workers and one job at a time, there's no errors and this:

[jszuminski]

@davidseverwright we've recently merged a new PR (#422) where we replaced the generic-pool package with tarn.

This could be a possible fix of this issue. Could you please try out the latest version (from the master branch or from the npm [email protected]) and let us know whether it solved it for you?

[davidseverwright]

Hi, thanks for looking into this.

It seems a lot better (less errors) but there are still errors. The log is attached.
This was run with 4 workers, and 4 simultaneous requests. It's still all OK with one request at a time.

hces-errors.txt

[PaulDalek]

Hi @davidseverwright

Thank you for all the information. We released v3.0.2 recently. Could you check how the server performs on your side with this version?

[davidseverwright]

Hi,

I just tested this with Highcharts v11.2 and export server v3.0.3, and got the same errors.
hces.log

[shawnwall]

noting we've been seeing increased 400's against the official highcharts export server as well.

[anantjagania]

Facing the same issue. The app requesting charts from server constantly gets 400s.

Found this in error logs.

Wed Mar 27 2024 22:31:41 GMT+0000 [error] - [export] Error encountered during export: Error: Rasterization timeout Wed Mar 27 2024 22:31:41 GMT+0000 [error] - [export] Error encountered during export: Error: Rasterization timeout Wed Mar 27 2024 22:31:42 GMT+0000 [error] - [export] Work: b8596f4cbb744301ba483c3fca412341 could not be completed, sending: In pool.postWork: Error trying to perform puppeteer export: In pool.postWork: Error: Rasterization timeout.

[dmytromakohin]

We have the same issue.

Sometimes after exporting of images via highcharts it redirects to "not working" page.

Will you fix this issue or we need to find alternative to highchart export?

It is very critical for us to have stable export on website which always works.

[jszuminski]

Most likely fixed by #521 in v4.0.0 (currently on the master branch).