Add retry loop into the scheduler handling resubmitting when running state failed

Add retry loop into the scheduler handling resubmitting when running state failed #18194