[Self-Host] RangeError: Invalid string length

[shaxiaozz]

environment:
firecrawl: main-branch
docker: 24.0.7
dify: 0.10.2

Describe the Issue
I am crawling a website with a lot of data. I am not using cURL to crawl, but dify to crawl.

I found that when the number of crawled pages reaches about 15,000 pages, the firecrawl-api-1 container will report an error "ERROR - RangeError: Invalid string length". As shown in the following log::
[2024-10-30T03:13:29.584Z]ERROR - RangeError: Invalid string length

And the dify front end will also report an error 500, as shown in the following screenshot:

I don't know where the problem is. I tried to adjust the --max-old-space-size parameter in the "apps/api/docker-entrypoint.sh" file to --max-old-space-size=30720

I also found that after firecrawl-api-1 reported an error, firecrawl-worker-1 continued to crawl until the crawl was completed. And I observed that the queue webui did not report any errors. (The website crawled a total of 16793 pages)

Expected Behavior
After firecrawl-worker-1 finishes crawling, firecrawl-api-1 responds to requests normally.