Skip to content
This repository has been archived by the owner on Apr 1, 2023. It is now read-only.

Add option to skip storing job args in status key #155

Open
ezekg opened this issue Feb 27, 2020 · 1 comment
Open

Add option to skip storing job args in status key #155

ezekg opened this issue Feb 27, 2020 · 1 comment

Comments

@ezekg
Copy link

ezekg commented Feb 27, 2020

I recently ran some diagnostics against my Sidekiq instance using RedisInsight and was shocked to see I had >1GB of data for the sidekiq:status:* keyspace. This is 10x larger than any other keyspace in my Redis database. Worth noting that I process ~1MM jobs a day that utilize sidekiq-status, and I have a status expiry of 3 days.

Upon inspection, the large keyspace was because this lib is storing the job's args for display purposes. In my case, this included potentially large JSON payloads.

I actually don't use the UI for Sidekiq, so this was needlessly wasting space and I feel like an option to disable this behavior would be beneficial for situations like this.

For now, I implemented the following monkey-patch in an initializer:

class Sidekiq::Status::ClientMiddleware
  def display_args(msg, queue)
    nil
  end
end

Thoughts?

@ezekg
Copy link
Author

ezekg commented Feb 27, 2020

And as an aside: for anybody else affected by this, here's a small rake task for cleaning up args for the current keyspace, to be used after applying the above monkey-patch:

# frozen_string_literal: true

desc 'clean up sidekiq-status args'
task sidekiq_status_args_cleanup: :environment do
  redis = Rails.cache.redis

  redis.with do |conn|
    conn.scan_each(match: 'sidekiq:status:*') do |key|
      puts "Clearing args: #{key}"

      conn.hdel key, :args
    end
  end
end

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant