How to tell bull queue that the worker is running the job, and not stalled

[vickyRathee]

I have long processing crawling job, it may take hours or days to complete a job.

const queueSettings: AdvancedSettings = {
  lockDuration: 60 * 1000, // Key expiration time for job locks. 5 minutes
  lockRenewTime: 30 * 1000, // Interval on which to acquire the job lock. 1 minutes
  stalledInterval: 60 * 1000, // How often check for stalled jobs (use 0 for never checking).
  maxStalledCount: 1, // Max amount of times a stalled job will be re-processed.
  guardInterval: 5000, // Poll interval for delayed jobs and added jobs.
  retryProcessDelay: 5000, // delay before processing next job in case of internal error.
  backoffStrategies: {}, // A set of custom backoff strategies keyed by name.
  drainDelay: 5, // A timeout for when the queue is in drained state (empty waiting for jobs).
};

const options: QueueOptions = {
  settings: queueSettings,
};

const jobsQueue = new Queue("Jobs", REDIS_URL, options);

I am reporting the progress { } after each request in loop and extended the stalledInterval to 60s, still the job is stalled after around a minute or two while the worker is running the job and no errors.

const urls = ["https://example.com", ""https://example.com".... ];
const data = { total: urls.length, processed: 0 , failed: 0, succeeded: 0};

for (let i = 0; i <urls .length; i++){
   data.processed += 1;
   // Crawl
   // Extract
   // Upload

  data.succeeded += 1;

  // Report progress
  job.progress(data).catch((err) => {
    console.error(err);
  });
}

[manast]

Are you processing the jobs manually like this: https://github.com/OptimalBits/bull/blob/develop/PATTERNS.md#manually-fetching-jobs or using the standard processor function?
You normally do not need to change the stalledInterval setting, and it is better that you do not do it, it has nothing to do with the time it takes for your job to complete. In any case, it is recommended to break your large jobs into smaller jobs, by definition a job can fail, would you lose several days of processing if it fails right at the end? By breaking into smaller jobs you do not have this risk and also you get better visibility.

[vickyRathee]

No, I am processing the jobs normally.
But, I have a custom script to execute via vm2. The script is CPU intensive and has nested loop.

Here is one example, which takes 40-60 seconds to Generate URL in nested loop > remove duplicates > write to input.csv

const input = ['url1', 'url2'....] // 70 URLs
const location = ['11001', '11002'....] // 3220 Zipcodes

const data = [];
    input.forEach(row => {
        locations.forEach(location => {
            data.push({ ...row, url: `${row.url}/?zip=${location }` });
        })
    });

// Remove duplicates

// fs.writeToFile

But it get stalled, here the logs.
The new duplicate job is started, while this one is running fine on worker

2022-02-20T08:01:00.267Z     INFO        Worker id: 12
2022-02-20T08:01:00.268Z     INFO        Job id: 2342 
2022-02-19T08:01:01.735Z	INFO	Compiled vm2 script
2022-02-19T08:01:02.345Z	INFO	Running vm2 script...
2022-02-19T08:01:40.815Z	INFO	Script completed successfully
2022-02-19T08:02:01.304Z	WARN	1280 duplicates input skipped
2022-02-19T08:02:01.122Z	INFO	Total inputs: 256400
2022-02-20T08:02:01.134Z     ERROR     **Job id: 2342 stalled**
2022-02-20T08:02:01.174Z     INFO        Running crawler...
2022-02-20T08:02:02.114Z     INFO        Running request 1 of 256400
....
...

[manast]

You can either break the loop so that you return to the event loop regularly or use a "sandboxed" bull processor.

[manast]

Stalled jobs do not make workers stop processing jobs. First you need to understand how nodejs event loop works, you should not keep it busy for a long time. There are several techniques to break a heavy loop into smaller chunks and return to the event loop between chunks.

[vickyRathee]

Any example link or docs for that to keep heavy loop into smaller chunk? I just need a way to tell the queue to not consider my job is stalled until timeout option set, which I have tried to extend to 2 minutes as well.

lockDuration: 60 * 2 * 1000, // Key expiration time for job locks. 2 minutes
lockRenewTime: 60 * 1000, // Interval on which to acquire the job lock. 1 minutes
stalledInterval: 60 * 2 * 1000, // How often check for stalled jobs (use 0 for never checking).

[SidSethi]

@vickyRathee I recently experienced this same issue (#2332)
I solved it by setting maxStalledCount = 0. I think you can also accomplish this by setting lockDuration to a value higher than your job duration, but I also did not want a stalled job to ever be re-processed so doing both should be safe

[vickyRathee]

Thanks @SidSethi I will try that.

[sibelius]

did this config worked?