Using Sidekiq Web to delete enqueued jobs - doesn't remove lock #14
Comments
|
Gemfile: |
|
@eldad87 @eldadvcita Thank you for the report. Unfortunately, I can not reproduce this bug. Here is my setup: ~ ruby -v
ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]
~ rails -v
Rails 6.0.3.4Redis is flushed: ~ redis-cli FLUSHDB
OKDummy rails application with ~ rails new test-activejob-uniqueness -G -O -M -P -T --api --skip-spring --skip-listen --skip-bootsnap --quiet \
&& cd test-activejob-uniqueness \
&& bundle add sidekiq activejob-uniqueness \
&& rails g job my_job \
&& echo "require 'sidekiq/web'\nRails.application.routes.draw { mount Sidekiq::Web => '/' }" > config/routes.rb \
&& sed -i '2 a\ \ unique :until_and_while_executing' app/jobs/my_job.rb \
&& sed -i '1 a\ \ config.active_job.queue_adapter = :sidekiq' config/environments/development.rb \
&& bundle exec rails sAt this point http://localhost:3000/queues shows no queues Running console ( :001 > MyJob.perform_later(123)
=> #<MyJob:0x000055a288263740 @arguments=[123], @job_id="4a37e212-329f-4d9c-baff-6d036ece87e2", @queue_name="default", @priority=nil, @executions=0, @exception_executions={}, @lock_strategy=#<ActiveJob::Uniqueness::Strategies::UntilAndWhileExecuting:0x000055a28823b3f8 @lock_key="activejob_uniqueness:my_job:4951b9977632a52fcd6f0cc65c57bb33", @lock_ttl=86400000, @on_conflict=:raise, @job=#<MyJob:0x000055a288263740 ...>, @runtime_lock_ttl=86400000, @on_runtime_conflict=:raise>, @provider_job_id="2d9274e00d2d2df1e582876a">
# The first job is enqueued successfully. I can see it in Sidekiq Web UI in the "default" queue
:002 > MyJob.perform_later(123)
Traceback (most recent call last):
1: from (irb):2
ActiveJob::Uniqueness::JobNotUnique (Not unique MyJob (Job ID: 1036d60c-a95f-489b-b816-bcd5c915d49c) (Lock key: activejob_uniqueness:my_job:4951b9977632a52fcd6f0cc65c57bb33) [123])
# The second attempt to enqueue the same job fails because it is not unique (as expected)
# Now I'm deleting exact job in the "default" queue via Sidekiq Web UI and making another attempt to enqueue the job:
:003 > MyJob.perform_later(123)
=> #<MyJob:0x000055a287f25c18 @arguments=[123], @job_id="36e28acf-241e-49da-894f-9320e874c9e5", @queue_name="default", @priority=nil, @executions=0, @exception_executions={}, @lock_strategy=#<ActiveJob::Uniqueness::Strategies::UntilAndWhileExecuting:0x000055a287f24e30 @lock_key="activejob_uniqueness:my_job:4951b9977632a52fcd6f0cc65c57bb33", @lock_ttl=86400000, @on_conflict=:raise, @job=#<MyJob:0x000055a287f25c18 ...>, @runtime_lock_ttl=86400000, @on_runtime_conflict=:raise>, @provider_job_id="e8045a3251f1a9f70cceb2e8">
# The job is enqueued and I can see it in Sidekiq Web UI
:004 > MyJob.perform_later(123)
Traceback (most recent call last):
1: from (irb):4
ActiveJob::Uniqueness::JobNotUnique (Not unique MyJob (Job ID: 3fcc9f6f-bb3f-4fc3-8a6e-4f13968a7c37) (Lock key: activejob_uniqueness:my_job:4951b9977632a52fcd6f0cc65c57bb33) [123])
# Another attempt is prevented (as expected)
# This time I delete the "default" queue completely via Sidekiq Web UI and making another attempt to enqueue the job:
:005 > MyJob.perform_later(123)
=> #<MyJob:0x000055a2877d2fb0 @arguments=[123], @job_id="4dfe2809-f98e-4f2d-91cf-4018df093c54", @queue_name="default", @priority=nil, @executions=0, @exception_executions={}, @lock_strategy=#<ActiveJob::Uniqueness::Strategies::UntilAndWhileExecuting:0x000055a2878036d8 @lock_key="activejob_uniqueness:my_job:4951b9977632a52fcd6f0cc65c57bb33", @lock_ttl=86400000, @on_conflict=:raise, @job=#<MyJob:0x000055a2877d2fb0 ...>, @runtime_lock_ttl=86400000, @on_runtime_conflict=:raise>, @provider_job_id="04921b021bb84f097e991c01">
# And it is enqueued successfullyCould you please try to reproduce the bug with this dummy application? |
|
Just FYI, we have a similar issue where the lock remains intact even though there are no jobs whatsoever either queued up or processing. Haven't been able to pinpoint the source of the stale lock yet, though, but it happens randomly yet almost daily. My hunch is that restarting the Heroku worker dynos does not properly cause the locks to be released. I have no hard evidence to prove this, but it often seems to coincide chronologically with our nightly reboots. |
|
@jarkko I might be experiencing a similar issue. How do you know if a lock is still "in tact". Is there a method call? |
|
@Aryk I don't remember exactly anymore, but I believe you know at least because you're getting the |
|
Dear fellows. So i had to manually go over all Job classes and use the .unlock method. Any ideas on how to release the lock when deleting jobs from the queue? |
|
Not a real solution, but using |
|
In case when the worker process is killed (e.g. My suggestion is to set a custom TTL per type of a job. Let's assume you know that the average job processing time is 5 seconds. Set the a custom TTL of 10 minutes then. In case of killed worker, there will be up to 10 minutes delay before the job will be unlocked automatically. But be careful with setting small values for the TTL. The queue latency should also be taken into account. If the queue is not utilized in time and its latency is over the TTL, jobs duplicates might appear. class MyJob < ActiveJob::Base
unique :until_executing, lock_ttl: 10.minutes
def perform(args)
# work
end
end |
Wow this was the great command it has worked for me. |
|
Same issue, manually deleting a job from the list of Scheduled Jobs in Sidekiq UI does not remove the lock. :( |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hey,
Using Sidekiq Web to delete enqueued jobs, lock remain active.
Steps to reproduce:
until_and_while_executingExpected result: Should be able to enqueue the job
Actual result: I recieve an error
Not unique ClassNameJob (Job ID: ce10cf03-d2c4-4369-a4ec-5f50316acf66) (Lock key: activejob_uniqueness:..The text was updated successfully, but these errors were encountered: