Blocking web requests until a job is complete, then responding with the response.

[bradgessler]

To put my questions into better context, the problem I'm trying to solve is this:

I have a /screenshots/:uid.png?url=http://example.com/ end point. Note that it's generating a png image (this is an important fact later). This end point queues a very heavy job that fires up a headless Chromium instance, takes a screenshot of that webpage, and stores the PNG in ActiveStorage.

If this was an HTML request, I could just use Turbo to show a "Loading" message, then update the page via ActionCable with the ActiveStorage image URL when it's done. The problem is I need to serve up PNG requests, which have no concept of ActionCable, HTML, etc.

Here's how I want to address this within PNG formats:

1. Connection A - Client initially connects to /screenshots/:uid.png?url=http://example.com/, which kicks off the screenshot job. I then want to keep the connection open to the client and wait for the job to complete before closing it. GoodJob will notify when the job is complete so the response can be written.
2. Connection B - Another client connects to that same end-point before the job is finished. Using GoodJob's concurrency control, I can detect this state. When this state is detected I want to keep the Puma server connection open and setup a connection to GoodJob's/pgs Postgres to receive a message when the Job is done processing.
3. When the job is finished, I want it to broadcast this fact so that Connection A and B can write the response to the client and close the Puma connections.

Here's roughly what I want the resulting code to look like in this world (see the format.png block):

class Links::CardsController < ApplicationController
  layout false

  before_action :assign_link, :assign_image

  def show
    respond_to do |format|
      format.html { render @scene, layout: false }
      format.png do
        # TODO: Throw the image on a background queue if its not generated, then
        # serve up a processing image (or status of sorts).
        if @image.attached?
          redirect_to rails_blob_path @image
        else
          process_and_wait @link.async("assign_#{image}").enqueue do
            redirect_to rails_blob_path @image
          end
        end
      end
    end
  end

  protected
    def image = :open_graph_image

    def assign_link
      @link = Link.find(params[:link_id])
    end

    def assign_image
      @image = @link.send image
    end

    def process_and_wait(job)
       # Check if the job is enqueued (or not).
       # If it's queued, give Puma/Rack whatever it needs know the response is going to be async.
       yield # When the job is finished, yield the block so the server can respond.
       # Do whatever cleanup is necessary for the subscriber
       
       # TODO: Handle the "else" condition that queues the job and calls this method again so it can wait.
    end
end

Ok, with that here are a few questions:

Has anybody already implemented this?

I always start with this question since MIT licenses OSS that are already implemented is usually the easiest way to solve a problem. 😄

Is GoodJob the right tool for the job?

I should probably be doing this with BEAM & Elixir, but a lot of this is already implemented in Rails so here I am. What other libraries or frameworks should I be looking at written in Ruby that could make this work?

I realize this starts bumping into the idea of "just deploy a pub/sub server", but I am targeting the same scale as GoodJob where I want small-to-medium-size apps to do this over PG so Moar Infra doesn't have to be setup.

Does GoodJob broadcast all job state changes via Notify?

From what I can tell reading through the source code, it doesn't. It looks like GoodJob uses the notification mechanism to inform the workers that there's a new job available to process (not other state changes). I mostly want to confirm this behavior to make sure I'm not missing something from the source code.

Should use GoodJob's Notify mechanism and channel for this problem or publish all of this into its own namespace?

I can always setup a new notification mechanism that publishes job statuses to a different channel, but broadcasting state changes over the channel for a job seems to make a lot of sense if this project would want to go in that direction. Alternatively an extension could be built, similar to the concurrency extension, that makes this optional. I think my only issue with this being an extension are things getting dicey if it has to integrate with the code GoodJob models.

What should I be worried about with respect to Ruby concurrency, Puma, and GoodJob if I do this?

I haven't done a lot of lower-level concurrency work in Puma, Ruby, or GoodJob. Any pointers on potential problems I'd run into while trying to solve this problem would be helpful.

[bensheldon]

I'll speak from my perspective on GoodJob: yes, I think we could make this happen. The pieces here, speaking from the library perspective are:

• Be able to ask GoodJob for a job record that has a given job_class, arguments, and maybe some other things specific to your enqueue that's like "find me this object in the job queue, if it exists". This is needed if that other request comes in for example.com. Maybe there is conceptual overlap here with concurrency controls?
• Have some sort of GoodJob.wait(job/active_job, job_id) that will hold until that job finishes (and then it could return true or false, or maybe even raise the exception; one thing to think about here is should it wait until the job completely finishes, or are there checkpoints like a single execution errored, but the job will still be retried further to think about with the interface).
• The simplest option for such an imagined GoodJob.wait would be to poll the database, the more complex would be to use LISTEN/NOTIFY. GoodJob doesn't currently broadcast job state to NOTIFY, but it could. This should probably have a separate channel (probably able to subscribe to a specific job id), and it's own custom Notifier object because the current Notifier object is designed to run in a background thread and like distribute messages to different objects, whereas I think you want to just run a LISTEN loop on the current thread until it pops.

Those are just my ideas. I'd love to know if other GoodJob users or anyone else has thoughts on this. I think it would be pretty easy to get to a working prototype (it's all just active record), but for longterm I'd want to try to have interfaces so the Active Record stuff isn't part of GoodJob's public interface.

[j-manu]

If you are using Puma, this blocks the request thread and a DB connection until the request completes. Is that a tradeoff you are willing to make?

If you are using falcon, it's relatively trivial to accomplish this. You don't even need anything special from good_job. You can just poll the status in a loop.