Support for Job Middleware

[sled]

We're currently using Sidekiq as job processor but would like to move to GoodJob.

We do use the Sidekiq middleware for various things like preserving the I18n.locale, setting the CurrentAttributes (https://github.com/sidekiq/sidekiq/blob/main/lib/sidekiq/middleware/current_attributes.rb) and keep track of distributed tracing IDs.

Is there a similar concept in GoodJob to hook into the enqueueing / dequeuing with the ability to store custom metadata beside the job's arguments? Because I want to avoid having to pass the metadata (such as request ID, tracing ID) as job arguments.

[bensheldon]

Some of those (timezone, i18n locale) are already included in the Active Job.

For everything else, I think you should do this as an ActiveJob extension for serializing data into the job record. I'm happy to work through what is needed because nearly everything is possible by extending Active Job without adding hooks to GoodJob... but I want to know if it isn't and address that.

I would design something generic that could be used like this: MyJob.set(foobar: optional_foobar_override).perform_later.... and if that looks right, we could collaborate on one specifically for current attributes.

module FoobarExtension
  extend ActiveSupport::Concern

  included do
    prepend Prepends
    attr_accessor :foobar
  end

  module Prepends
    # `options` are passed into `MyJob.set(options)...`
    def enqueue(options = {})
      if options.key?(:foobar)
        self.foobar = options[:foobar]
      else
        self.foobar = Thread.current[:foobar] # some implicit / thread-local state
      end

      super
    end

    def serialize
      super.tap do |job_data|
        job_data["foobar"] = foobar unless foobar.nil?
      end
    end

    def deserialize(job_data)
      super

      self.foobar = job_data["foobar"]
    end
  end
end


class ApplicationJob
  include FoobarExtension
end

[sled]

thanks for the amazing reply! I think this should work for propagating the current attributes, I think I'll have to do some experimentation.

I also noticed your PR over at datadog (DataDog/dd-trace-rb#1633), do you think it would be of any benefit to add spans for good job's instrumentation? To get a deeper insight into latency, query times etc.?

For Sidekiq we added a distributed tracing middleware, so that if the job was caused by a request we can continue the trace in the job:

Maybe this could be done for ActiveJob in general as well using the available callbacks like around_perform

[gap777]

@bensheldon Has anyone already crafted AJ extension to handle CurrentAttributes?

[bensheldon]

Not yet! I think I'd accept it if someone did. I like an interface like:

class ApplicationJob < ActiveJob::Base
  persist_current_attributes :Current
  # or maybe more targetted constant => attributes
  persist_current_attributes Current: [:user]
end