Sidekiq Changes | Sidekiq Pro Changes | Sidekiq Enterprise Changes
Please see sidekiq.org for more details and how to buy.
- Move from SHA1 to SHA256 internally. Unique locks are now SHA256 but only take 10% more space due to using a more efficient encoding (base64 vs hex).
- Sidekiq::Web internal refactoring [#6532]
- Limiter factory methods now raise ArgumentError if you pass them a block [#6512]
- Call
Process.warmup
before fork in sidekiqswarm, useRUBY_DISABLE_WARMUP=1
to disable [#6279] - Disable health checks if running sidekiqswarm, we can't use one port to monitor many children
- Activate unique server middleware in client-mode so uniqueness works for
perform_inline
[#6460] - Add
Sidekiq::Enterprise.gem_version
API
- Fix healthcheck when defined in config YML [#6352]
- Allow rate limiting to use Redis Cluster [#6288]
- Remove
base64
gem dependency
- Revert #6288 as this requires Sidekiq 7.3 [#6324]
- Adjust the unlimited rate limiter to work as a points limiter too [#6301]
- Limiters can now use a clustered Redis client [#6288]
- Limiters now have attr_readers for all static config elements [#6259]
- Handle fractional values for SIDEKIQ_COUNT when containers have fractional CPU allocations,
e.g.
SIDEKIQ_COUNT=2.5
will create 2 Sidekiq processes. [#6244]
- Concurrent rate limiter raising ReadTimeoutError? Work around redis/redis#11732 [#6188]
- Add
within_limit(used: 1)
option towindow
andbucket
rate limiters. You can adjust the number of points used by a call performing batch operations [#6146] - Use HWIA when scheduling periodic ActiveJobs, for compatibility [#6099]
- Kubernetes health check can be enabled through Sidekiq's config YML. The full binding address can also be configured, not just port. Examples:
---
health_check: 127.0.0.1:8111 # static
health_check: <%= ENV["SIDEKIQ_HEALTH_BINDING"] %> # dynamic!
config.health_check("127.0.0.1:8111")
- Add support for a Kubernetes liveness / health check port. Start it with
config.health_check(port = 7433)
[#5923] - Add missing points view [#6016]
- Add Polish translations
- Please note that license credentials are required when running in production. We recommend configuring credentials with Bundler.
- Fix hash mutation race condition in rate limiter autoloading [#5908]
- NEW Points-based rate limiter popular with GraphQL endpoints at Shopify, GitHub, et al. Thanks to Thad Sauter of NexHealth for contributing the initial skeleton. [#5757]
- NEW Test helper to verify periodic job registration block [#5832]
require "sidekiq-ent/periodic/testing"
CRON_BLOCK = ->(mgr) { mgr.register("0 * * * * *", "SomeJob") }
ct = Sidekiq::Periodic::ConfigTester.new
ct.verify(&CRON_BLOCK) # => raises ArgumentError, invalid crontab syntax
- Periodic jobs may now be ActiveJobs [#5902]
- Refactor rate limiter codebase to use
autoload
- Refactor concurrent and bucket rate limiter data model to be cluster-friendly [#5800]
- Fix mutable job arguments when rescheduling an OverLimit [#5859]
- Tweak concurrent rate limiter to minimize spurious ReadTimeoutErrors [#5838]
- Fix redis-client API usage which could result in stuck Redis connections [#5823]
- Revert unique impl which required Redis 7.0 [#5793]
- Fix spurious "Uh oh" messages with
sidekiqswarm
[#5801]
- Remove usage of
replicate_commands
Redis directive, default in 5.0, gone in 7.0 - Fix issue with rate limiter connection pool [#5752]
- Unique middleware now prints the JID holding the lock if there is a duplicate [#5736] NOTE: Unique locks set with older versions (<7.0.4) will not work with newer versions.
- Allow user to define the context used to calculate unique locks, see the Unique Jobs wiki page [#5544]
- Smarter connection pool sizing for rate limiters [#5685]
- Fix crash in graceful restarts [#5667]
- Fix spurious ReadTimeoutError in concurrent rate limiter [#5611]
- Fix eager connection to Redis [#5606]
- Componentize and capsulize Enterprise functionality for Sidekiq 7
- Remove bucket history graph from Web UI
- Rename "Cron" tab to "Periodic" [#5590]
- Add DE locale
- Adjust rate limiters to lazy initialize, avoiding connection issues when forking preloaded app code [#5535]
- Remove Redis 4.8.0 deprecation warnings
- Fix crash with empty periodic data [#5374]
- Per the 2.0 upgrade notes, Sidekiq Enterprise will stop if you do not
have a licensed username configured on startup. It will extract it from
the Bundler environment if possible or look for SIDEKIQ_ENT_USERNAME.
SIDEKIQ_ENT_USERNAME=abcd1234 bundle exec sidekiq
- Internal refactoring for Sidekiq 6.5.
- Requires Sidekiq 6.5, Pro 5.5.
- Fix multi/pipe deprecation in redis-rb 4.6
- Leader now elects more often, to minimize missed cron jobs
- Fix periodic jobs missing the "fallback" hour during DST changeover [#5049]
- Remove jQuery usage in UI tabs
- Pass exception to rate limiter backoff proc [#5024]
- Fixes for leaky and unlimited limiters [#4809, #4869]
- Invalid leaders now immediately step down [#4950]
- Web UI now displays "next run time" in the specified timezone [#4833]
- Fix swarm memory monitoring on BSDs
- Periodic job timezone fix [#4796]
- Support configurable timezones for periodic jobs [#4749]
- Handle edge case leading to negative expiry in uniqueness [#4763]
- Add new leaky bucket rate limiter [#4414] This allows clients to burst up to X calls before throttling back to X calls per Y seconds. To limit the user to 60 calls per minute:
leaker = Sidekiq::Limiter.leaky("shopify", 60, :minute)
leaker.within_limit do
...
end
See the Rate Limiting wiki page for more detail.
- Rate limiters may now customize their reschedule count [#4725]
To disable rate limit reschedules, use
reschedule: 0
.
Sidekiq::Limiter.concurrent("somename", 5, reschedule: 0)
- Allow filtering by name in the Rate Limiter UI [#4695]
- Add IT locale
- The Sidekiq Pro and Enterprise gem servers now
bundle install
much faster with Bundler 2.2+ [#4158] - Now that ActiveJobs support
sidekiq_options
, add support for uniqueness in AJs [#4667]
- Add optional app preload in swarm, saves even more memory [#4646]
- Fix incorrect queue tags in historical metrics [#4377]
- Move historical metrics to use tags rather than interpolating name [#4377]
sidekiq.enqueued.#{name} -> sidekiq.queue.size with tag queue:#{name}
sidekiq.latency.#{name} -> sidekiq.queue.latency with tag queue:#{name}
- Remove
concurrent-ruby
gem dependency [#4586] - Add systemd
Type=notify
support for swarm [#4511] - Length swarm's boot timeout to 60 sec [#4544]
- Add NL locale
- Periodic job registration API adjusted to avoid loading classes in initializer [#4271]
- Remove support for deprecated ENV variables (COUNT, MAXMEM_MB, INDEX) in swarm code
- Except for the newly required credentials, Sidekiq Enterprise 2.0 does not have any significant migration steps.
- Sidekiq Enterprise must now be started with valid license credentials. [#4232]
- Call
GC.compact
if possible in sidekiqswarm before forking [#4181] - Changes for forward-compatibility with Sidekiq 6.0.
- Add death handler to remove any lingering unique locks [#4162]
- Backoff can now be customized per rate limiter [#4219]
- Code formatting changes for StandardRB
- Fix excessive lock reclaims with concurrent limiter [#4105]
- Add ES translations, see issues #3949 and #3951 to add your own language.
- Require Sidekiq Pro 4.0 and Sidekiq 5.2.
- Refactor historical metrics API to use revamped Statsd support in Sidekiq Pro
- Add a gauge to historical metrics for
default
queue latency [#4079]
- Add PT and JA translations
- Fix elapsed time calculations to use monotonic clock [#4000, sj26]
- Fix edge case where flapping leadership would cause old periodic jobs to be fired once [#3974]
- Add support for sidekiqswarm memory monitoring on FreeBSD [#3884]
- Fix Lua error in concurrent rate limiter under heavy contention
- Remove superfluous
freeze
calls on Strings [#3759]
- NEW FEATURE Rolling restarts - great for long running jobs!
- Adjust middleware so unique jobs that don't push aren't registered in a Batch [#3662]
- Add new unlimited rate limiter, useful for testing [#3743]
limiter = Sidekiq::Limiter.unlimited(...any args...)
- Fix crash in rate limiter middleware when used with custom exceptions [#3604]
- Show process "leader" tag on Busy page, requires Sidekiq 5.0.2 [#2867]
- Capture custom metrics with the
save_history
API. [#2815] - Implement new
unique_until: 'start'
policy option. [#3471]
- Fix broken Cron page in Web UI [#3458]
- Remove dependency on the algorithms gem [#3446]
- Allow user to specify max memory in megabytes with SIDEKIQ_MAXMEM_MB [#3451]
- Implement logic to detect app startup failure, sidekiqswarm will exit rather than try to restart the app forever [#3450]
- Another fix for doubly-encrypted arguments [#3368]
- Fix encrypted arguments double-encrypted by retry or rate limiting [#3368]
- Fix leak in concurrent rate limiter, run this in Rails console to clean up existing data [#3323]
expiry = 1.month.to_i; Sidekiq::Limiter.redis { |c| c.scan_each(match: "lmtr-cfree-*") { |key| c.expire(key, expiry) } }
- Fix issue with census startup when not using Bundler configuration for source credentials.
- Add new web authorization API [#3251]
- Update all sidekiqswarm env vars to use SIDEKIQ_ prefix [#3218]
- Add census reporting, the leader will ping contribsys nightly with aggregate usage metrics
- No functional changes, require latest Sidekiq and Sidekiq Pro versions
- Upgrade encryption to use OpenSSL's more secure GCM mode. [#3060]
- Fix multi-process memory monitoring on CentOS 6.x [#3063]
- Polish the new encryption feature a bit.
- BETA New encryption feature which automatically encrypts the last argument of a Worker, aka the secret bag.
- Fix issue causing some minutely jobs to execute every other minute.
- Log a warning if slow periodic processing causes us to miss a clock tick.
- Periodic jobs could stop executing until process restart if Redis goes down [#3047]
- Add API to check if a unique lock is present. See [#2932] for details.
- Tune concurrent limiters to minimize thread thrashing under heavy contention. [#2944]
- Add option for tuning which Bundler groups get preloaded with
sidekiqswarm
[#3025]
SIDEKIQ_PRELOAD=default,production bin/sidekiqswarm ...
# Use an empty value for maximum application compatibility
SIDEKIQ_PRELOAD= bin/sidekiqswarm ...
- Multi-Process mode can now monitor the RSS memory of children and restart any that grow too large. To limit children to 1GB each:
MAXMEM_KB=1048576 COUNT=2 bundle exec sidekiqswarm ...
- NEW FEATURE Multi-process mode! Sidekiq Enterprise can now fork multiple worker processes, enabling significant memory savings. See the wiki documentation for details.
- More precise gemspec dependency versioning
- NEW FEATURE Historical queue metrics, documented in the wiki [#2719]
- Window limiters can now accept arbitrary window sizes [#2686]
- Fix race condition in window limiters leading to non-stop OverLimit [#2704]
- Fix invalid overage counts when nesting concurrent limiters
- Fix crash in periodic subsystem when a follower shuts down, thanks to @justinko for reporting.
- Enterprise 1.x targets Sidekiq 4.x.
- Rewrite several features to remove Celluloid dependency. No functional changes.
- Fix
unique_for: false
[#2658]
- Enterprise 0.x targets Sidekiq 3.x.
- Fix racy shutdown event which could lead to disappearing periodic jobs, requires Sidekiq >= 3.5.3.
- Add new :leader event which is fired when a process gains leadership.
- Redesign how overrated jobs are rescheduled to avoid creating new jobs. [#2619]
- Fix dynamic creation of concurrent limiters [#2617]
- Add additional check to prevent duplicate periodic job creation
- Allow user-specified TTLs for rate limiters [#2607]
- Paginate rate limiter index page [#2606]
- Rework
Sidekiq::Limiter
redis handling to match global redis handling. - Allow user to customize rate limit backoff logic and handle custom rate limit errors.
- Fix scalability issue with Limiter index page.
- Fix typo which prevented limiters with '0' in their names.
- Fix issue where unique scheduled jobs can't be enqueued upon schedule due to the existing unique lock. [#2499]
Initial release.