Is it possible to have a parent batch wait for a child batch to complete before continuing?

[dimroc]

Hi there, I complex workflow that has a parent batch spawning multiple batches. Is it possible to have a batch wait on another batch?

Here's a simple repro of the idea below, and based on my testing, it doesn't happen when using GoodJob::Batch.enqueue.

class TestParentBatchJob < ApplicationJob
  def perform(batch, _params)
    if batch.properties[:stage].nil?
      Rails.logger.info("TESTTEST TestParentBatchJob START: #{batch.properties}")
      batch.enqueue(stage: 1) do
        GoodJob::Batch.enqueue(
          on_success: TestChildBatchJob,
          dimi: 'sent from parent to child'
        )
      end
    elsif batch.properties[:stage] == 1
      Rails.logger.info("TESTTEST TestParentBatchJob Next Stage: #{batch.properties}")
      batch.enqueue(stage: 2) do
        TestIndividualJob.perform_later('sent from parent batch to 1 job')
      end
    end
  end
end

class TestChildBatchJob < ApplicationJob
  def perform(batch, _params)
    Rails.logger.info("TESTTEST TestChildBatchJob START: #{batch.properties}")
    sleep 10
    if batch.properties[:stage].nil?
      batch.enqueue(stage: 1) do
        TestIndividualJob.perform_later('sent from child batch to 1 job')
      end
    elsif batch.properties[:stage] == 1
      Rails.logger.info("TESTTEST TestChildBatchJob Next Stage: #{batch.properties}")
      batch.enqueue(stage: 2) do
        TestIndividualJob.perform_later('sent from child batch to 2 job')
      end
    else
      Rails.logger.info('TESTTEST TestChildBatchJob ACTUALLY FINISHED')
    end
  end
end

class TestIndividualJob < ApplicationJob
  def perform(text)
    Rails.logger.info("TESTTEST TestIndividualJob START: #{text}")
    sleep 10
    Rails.logger.info("TESTTEST TestIndividualJob: #{text}")
  end
end

[bensheldon]

GoodJob doesn't have a hierarchy between batches.

Unlike Sidekiq, GoodJobs batches are mutable: jobs can be added to a batch, and batches can be created (with jobs, that are performed) before the batch is marked as enqueued; and a batch that has previously finished can have new jobs added to it, and be marked as enqueued again which will retrigger callbacks.

In this example, wait to call enqueue on the pseudo-parent batch in the pseudo-child batch's batch callback. That would produce an equivalent effect I think.

I'm sorta reluctant to add hierarchy/relations between batches myself. Because GoodJob batches are mutable, it would be a lot of complex decisions to work though. And at the risk of being to candid, batches are a feature that has largely evolved from people wanting abstract or comparative solutions rather than concrete real-world scenarios that I find a lot more fun to solve for 😀

[dimroc]

Thanks for being responsive and thanks for the suggestion.

I was able to trigger a callback at the time expected by enqueuing the parent at the end of the child batch. However, this won't be a good solution if there are N child batches, because then we'd have to manually assess if each child is complete, which defeats the purpose of a batch. I will think on this more.

As for real-world scenarios, there are a few concrete examples people run into regularly we can solve for. For example, I currently want to fan out a few jobs to run in parallel for a file, let's call that FileBatchJob. Now, I've expanded to support folders. Ideally, I would run a single FolderBatchJob to run N FileBatchJobs. With this current approach, we'd have to do manual counting of whether the N FileBatchJobs are done, by enqueuing a pseudo-parent and counting if M/N == 1.

Below, I've incorporated your suggestion into the code:

class TestParentBatchJob < ApplicationJob
  def perform(batch, _params)
    if batch.properties[:stage].nil?
      Rails.logger.info("1. TESTTEST TestParentBatchJob START: #{batch.properties}")
      GoodJob::Batch.enqueue(
        on_success: TestChildBatchJob,
        parent_batch: batch,
        dimi: 'sent from parent to child'
      )
    elsif batch.properties[:stage] == 1
      Rails.logger.info("9. TESTTEST TestParentBatchJob Next Stage: #{batch.properties}")
      batch.enqueue(stage: 2) do
        TestIndividualJob.perform_later('sent from parent batch to 1 job')
      end
    end
  end
end

class TestChildBatchJob < ApplicationJob
  def perform(batch, _params)
    if batch.properties[:stage].nil?
      Rails.logger.info("2. TESTTEST TestChildBatchJob START: #{batch.properties}")
      sleep 10
      batch.enqueue(stage: 1) do
        TestIndividualJob.perform_later('sent from child batch to 1 job')
      end
    elsif batch.properties[:stage] == 1
      Rails.logger.info("5. TESTTEST TestChildBatchJob Next Stage: #{batch.properties}")
      batch.enqueue(stage: 2) do
        TestIndividualJob.perform_later('sent from child batch to 2 job')
      end
    else
      Rails.logger.info('8. TESTTEST TestChildBatchJob ACTUALLY FINISHED')
      parent_batch = batch.properties[:parent_batch]
      parent_batch.enqueue(stage: 1)
    end
  end
end

class TestIndividualJob < ApplicationJob
  def perform(text)
    Rails.logger.info("3. 6. 10. TESTTEST TestIndividualJob START: #{text}")
    sleep 10
    Rails.logger.info("4. 7. 11. TESTTEST TestIndividualJob: #{text}")
  end
end

[bensheldon]

I think I understand. The way I handle this is that I don't "enqueue" the batch until all of the jobs for a given stage of the batch exist. So if you want to have a sub-batch and wait for it don't (re-)enqueue the parent batch until the sub-batch has finished, and then enqueue a SignalJob/NoopJob to the parent batch when the child batch is done as part of its callback.

sorry, I'm mobile or else I'd give a code example.