How to parallelly run several crawlers? I get some errors!

[Ramin-Bateni]

How should I run several crawlers parallelly?

Currently, I run 2 crawler by this way:

await Promise.all([
  crawler1.run(),
  crawler2.run(),
]);

But I get this error:
ENOENT: no such file or directory, open 'D:\project-crawler\storage\request_queues\default\iMCIUbrRaOsVice.json']

if I comment one of the crawlers it works correctly:

await Promise.all([
  crawler1.run(),
  // crawler2.run(),
]);

Additionally, what is your suggested way of running crawlers parallelly?

[B4nan]

This is a bit more complex, you will need two configuration instances, and override how the internal state is stored - you will need to change the default RQ and KVS names to have dedicated folders for those.

Something like this could work, you might need to override more options:

const config1 = new Configuration({
  defaultRequestQueueId: 'rq1',
  defaultKeyValueStoreId: 'kvs1',
});

const config2 = new Configuration({
  defaultRequestQueueId: 'rq2',
  defaultKeyValueStoreId: 'kvs2',
});

const crawler1 = new CheerioCrawler({ ... }, config1);
const crawler2 = new CheerioCrawler({ ... }, config2);

await Promise.all([
    crawler1.run(),
    crawler2.run(),
]);