Missing the runs sometimes.

[tushar-meesho]

Describe the bug

My app is running on 20 machines, and I want scheduler to be executed once in every 5 minutes in any 1 machine by using shedlock.

Scheduler:

    @Scheduled(cron = "0 */5 * * * *")
    @SchedulerLock(name = "executeTaskLock",
            lockAtLeastFor = "1m",
            lockAtMostFor = "4m")
    public void executeTask() throws Exception {
        LockAssert.assertLocked();

        // task executes here which completed in around (400ms - 2s)
    }

Config:

    @Bean
    public JedisPool jedisPoolForSchedulerLocking() {

        JedisPoolConfig jedisPoolConfig = new JedisPoolConfig();
        jedisPoolConfig.setMaxTotal(2);
        jedisPoolConfig.setMaxIdle(2);
        jedisPoolConfig.setMinIdle(1);
        jedisPoolConfig.setMaxWaitMillis(500);

        return new JedisPool(jedisPoolConfig, {host}, {port}, {timeout}, {port}); 
    }

    @Bean
    public LockProvider lockProvider(@Qualifier("jedisPoolForSchedulerLocking") JedisPool jedisPool) {
        return new JedisLockProvider(jedisPool, "PRD");
    }

pom.xml

        <dependency>
            <groupId>net.javacrumbs.shedlock</groupId>
            <artifactId>shedlock-spring</artifactId>
            <version>4.42.0</version>
        </dependency>

        <dependency>
            <groupId>net.javacrumbs.shedlock</groupId>
            <artifactId>shedlock-provider-redis-jedis</artifactId>
            <version>4.42.0</version>
        </dependency>

Expected behavior

It should run at-least once every 5 mins without any run miss.

Actual behavior

I am seeing sometimes scheduler is missing the runs, for instance it runs at 12:25, 12:30 and then runs at 12:45, missed 2 runs here. Actual task completes in around (400ms - 3s), thus I have setup 1m and 4m as lockAtLeastFor and lockAtMostFor. config seems correct to me.

Doesn't know what's failing here exactly. As per official docs this can run more than once(which is fine for my use-case) but should never miss the runs.

grafana of tracking the runs (you can clearly see it has missed runs many times).

Stack Reference : https://stackoverflow.com/questions/77566840/shedlock-skipped-runs

[tushar-meesho]

@lukas-krecan Please check this

[lukas-krecan]

https://github.com/lukas-krecan/ShedLock#troubleshooting

[tushar-meesho]

I have tried the debug guide but no help so far.

1. I could only manage to upgrade to 4.46.0, as this is the latest version for shedlock-provider-redis-jedis available on maven repo : https://mvnrepository.com/artifact/net.javacrumbs.shedlock/shedlock-provider-redis-jedis.
2. I am using LockAssert as you can see in the code above.
3. How to on the debug logs exactly for shedlock specifically ?
4. I am using redis based on jedis client and I am also using lockAtleastFor = 1m as my scheduler runs in every 5 mins, and I have no issue if my scheduler does some extra runs, but it should not miss the runs at all

@lukas-krecan

[lukas-krecan]

Sorry, if you are not able to configure logging, I will not be able to help you

[lukas-krecan]

This issue looks similar #1611 (comment)

[tushar-meesho]

1. I have enabled the debug logs now, log does not show anything.
2. According to the issue mentioned above, it says that we need to set the spring.task.scheduling.pool.size to a higher value than 1, but that should not make the scheduler skip the runs, it should make it delay if 2 schedulers runs at the same time as we only have 1 thread, so 2nd task is going to wait untill 1st task gets executeed. In my case this is getting skipped entirely (without any execution).
3. Plus I have 2 scheduler in my service currently and nearly 10 machines in which app is running, if one machine is going to start other scheduler first, than other any machine from other 9 machine should run the shedlock scheduler for sure. But still seeing the misses. No Error/Debug logs at anywhere.