Lambda Issue

[tsrdatatech]

I deployed a lambda on AWS where I want to do some crawling. When I try to run it I get the following error:

{ "errorType": "Runtime.UnhandledPromiseRejection", "errorMessage": "Error: ENOENT: no such file or directory, mkdir '/var/task/storage'", "trace": [ "Runtime.UnhandledPromiseRejection: Error: ENOENT: no such file or directory, mkdir '/var/task/storage'", " at process.<anonymous> (file:///var/runtime/index.mjs:1131:17)", " at process.emit (node:events:527:28)", " at processEmit [as emit] (/var/task/node_modules/signal-exit/index.js:199:34)", " at emit (node:internal/process/promises:140:20)", " at processPromiseRejections (node:internal/process/promises:274:27)", " at processTicksAndRejections (node:internal/process/task_queues:97:32)" ] }

Now tbh I only suspect Crawlee could be the issue only because of the storage mentioned in the error, hoping some guidance here can help. For context I use the following settings in my crawlee.json file:

{ "persistStorage": false, "localDataDirectory": "/tmp", "writeMetadata": false }

The above to me would seem to say that nothing would be written to any path or dir, is that correct? Just for some testing I had added the localDataDirectory path just in case it was still writing something. Also note I do not write anything to any place within my code.

Thanks.

[B4nan]

How do you run the crawler?

cc @vladfrangu, is it possible the auto-purge would fail if the folder does not exist?

[B4nan]

I dont see anything weird, localDataDirectory is not a valid crawlee config option. It would have to be storageClientOptions: { localDataDirectory: '...' } if something.

[B4nan]

Well, I really think this is from the auto-purge...

[vladfrangu]

But that method is called only for when a request gets deleted, something that's not called when purging at boot 👀

[B4nan]

It looks like this is the only place that could do it, I dont see any other "unsafe" ensureDir or mkdir call. I am not saying it makes sense, but I don't see any other way this could happen.

[tsrdatatech]

Another thing to keep in mind regarding this is that when you deploy to AWS you automatically have a dir called var/task and in the error I posted it has storage listed there so perhaps it could be related to storage stuff besides the auto purge?

[tsrdatatech]

I just have a simple handler function that has:

await crawler.run(['https://www.durangolocal.news/', 'https://www.co.laplata.co.us/newslist.php', 'https://coloradosun.com/tag/la-plata-county/', 'https://www.newsbreak.com/channels/la-plata-county-co', 'https://www.durangotelegraph.com/', 'https://www.newsbreak.com/channels/durango-co', 'https://www.durangoherald.com/tags/breaking-news/', 'https://www.durangoherald.com/categories/local-news/']);

And the actual code for crawling is just:

`let links = [];
const crawler = new CheerioCrawler({
"maxRequestsPerCrawl": 50,
async requestHandler({ request, enqueueLinks, body }) {
links.push({'url': request.url, 'content': body});

    // Add all links from page to RequestQueue
    await enqueueLinks({
        "strategy": "same-domain",
        transformRequestFunction(req) {
            if (req.url.endsWith('.pdf')) return false;
            if (req.url.endsWith('.jpg')) return false;
            if (req.url.endsWith('.jpeg')) return false;
            if (req.url.endsWith('.png')) return false;
            return req;
        }
    });
}

});`

[B4nan]

I was asking how you run it, if you use the CLI or just do npm/yarn start. I believe the Apify CLI does create folders automatically, but I guess you are not using that?

[tsrdatatech]

correct I am not using that.

[tsrdatatech]

Just for some more context I created a container for my lambda and this issue disappeared. Not sure if it means it is the ensureDir being unsafe or not but it seems to not reproduce this error now using a container.

[vladfrangu]

Could you please provide a full code sample that we can test ourselves in a Lambda env to debug this and help you out 🙏

[tsrdatatech]

Yea sure, basically I used this:

import { CheerioCrawler, ProxyConfiguration } from 'crawlee';

const links = [];
const crawler = new CheerioCrawler({
    "maxRequestsPerCrawl": 50,
    async requestHandler({ request, enqueueLinks, log }) {
        log.info(request.url);
        links.push(request.url);
        // Add all links from page to RequestQueue
        await enqueueLinks({
            "strategy": "same-domain",
            "regexps": [/([./\-_]{0,1}(19|20)\d{2})[./\-_]{0,1}(([0-3]{0,1}[0-9][./\-_])|(\w{3,5}[./\-_]))([0-3]{0,1}[0-9][./-]{0,1})?/],
            'globs': [`${request.url}news/*`,
                      `${request.url}story/*`,
                      `${request.url}story/news/*`,
                      `${request.url}category/news/*`,
                      `${request.url}category/headline-news/*`,
                      `${request.url}*/news/*`],
            transformRequestFunction(req) {
                if (req.url.endsWith('.pdf')) return false;
                if (req.url.endsWith('.jpg')) return false;
                if (req.url.endsWith('.jpeg')) return false;
                if (req.url.endsWith('.png')) return false;
                return req;
            }
        });
    }
});

await crawler.run(['https://www.newsbreak.com/channels/la-plata-county-co']);

And I just added a crawlee.json that has this:

{
    "persistStorage": false
}

The lambda stuff is very generic usual lambda setup is all I used.

Let me know if this helps.