Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added metrics to Scheduler and track in Process state #984

Merged
merged 23 commits into from
Jul 4, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
b657d1f
added metrics
AndersGM Jun 13, 2023
3111a28
thread safe counters
AndersGM Jun 20, 2023
1f3ae5f
reset metrics after restart
AndersGM Jun 20, 2023
2960a86
specced both error and success in #task_observer
AndersGM Jun 20, 2023
30db43a
added doc
AndersGM Jun 20, 2023
78d8aba
include metrics in name
AndersGM Jun 20, 2023
042ed03
Merge branch 'main' into metrics
AndersGM Jun 20, 2023
17e5bd7
renamed and reverted
AndersGM Jun 25, 2023
02dae4a
Merge remote-tracking branch 'fork/metrics' into metrics
AndersGM Jun 25, 2023
afeae87
consistent naming
AndersGM Jun 25, 2023
2c3c21e
fixe broken current state
AndersGM Jun 25, 2023
6069cac
Merge branch 'main' into metrics
AndersGM Jun 25, 2023
822a0be
Merge remote-tracking branch 'origin/main' into metrics
bensheldon Jul 1, 2023
e9a26d2
Track Scheduler name and queues separately in stats and display
bensheldon Jul 1, 2023
28b99ad
Fix Scheduler tests
bensheldon Jul 1, 2023
6c9af51
Remove default value for state
bensheldon Jul 1, 2023
3ca81d4
Renamed `failed_` to `errored_`
bensheldon Jul 1, 2023
e59c8da
Track empty and unlocked executions too
bensheldon Jul 1, 2023
74ad726
Have stat totals add up consistently
bensheldon Jul 1, 2023
6fe747a
Ensure stats can be reported when shutdown
bensheldon Jul 1, 2023
169a23c
Don't add object to instances list until fully initialized
bensheldon Jul 1, 2023
0964768
Fix race condition in Scheduler#stats test
bensheldon Jul 1, 2023
b8f6f08
Rename "unlocked" to "unexecutable" and fix counting
bensheldon Jul 3, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions lib/good_job/metrics.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# frozen_string_literal: true
module GoodJob # :nodoc:
class Metrics
attr_reader :failed_count, :succeeded_count

def initialize
@failed_count = 0
@succeeded_count = 0
end

def increment_failed
@failed_count += 1
AndersGM marked this conversation as resolved.
Show resolved Hide resolved
end

def increment_succeeded
@succeeded_count += 1
end
end
end
11 changes: 11 additions & 0 deletions lib/good_job/scheduler.rb
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
require "concurrent/executor/timer_set"
require "concurrent/scheduled_task"
require "concurrent/utility/processor_counter"
require 'good_job/metrics'

module GoodJob # :nodoc:
#
Expand Down Expand Up @@ -88,6 +89,7 @@ def initialize(performer, max_threads: nil, max_cache: nil, warm_cache_on_initia
@executor_options[:name] = name

@cleanup_tracker = CleanupTracker.new(cleanup_interval_seconds: cleanup_interval_seconds, cleanup_interval_jobs: cleanup_interval_jobs)
@metrics = ::GoodJob::Metrics.new
create_executor
warm_cache if warm_cache_on_initialize
end
Expand Down Expand Up @@ -203,6 +205,12 @@ def task_observer(time, output, thread_error)
instrument("finished_job_task", { result: output, error: thread_error, time: time })
return unless output

if error
@metrics.increment_failed
else
@metrics.increment_succeeded
end

@cleanup_tracker.increment
if @cleanup_tracker.cleanup?
cleanup
Expand All @@ -222,6 +230,9 @@ def stats
max_cache: @max_cache,
active_cache: cache_count,
available_cache: remaining_cache_count,
succeeded_count: @metrics.succeeded_count,
failed_count: @metrics.failed_count,
total_count: @metrics.succeeded_count + @metrics.failed_count,
}
end

Expand Down
22 changes: 22 additions & 0 deletions spec/lib/good_job/scheduler_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,25 @@

expect(error_proc).to have_received(:call).with(an_instance_of(StandardError).and(having_attributes(message: 'oopsy'))).at_least(:once)
end

it 'increases failed job count' do
expect(performer).to receive(:next) do
THREAD_HAS_RUN.make_true
GoodJob::ExecutionResult.new(value: nil, unhandled_error: StandardError.new("oopsy"))
end

allow(error_proc).to receive(:call) do
ERROR_TRIGGERED.make_true
end

scheduler = described_class.new(performer)
scheduler.create_thread
sleep_until { THREAD_HAS_RUN.true? }
sleep_until { ERROR_TRIGGERED.true? }

expect(scheduler.stats.fetch(:failed_count)).to eq 1
expect(scheduler.stats.fetch(:succeeded_count)).to eq 0
end
end
end

Expand Down Expand Up @@ -177,6 +196,9 @@
max_cache: max_cache,
active_cache: 0,
available_cache: max_cache,
failed_count: 0,
succeeded_count: 0,
total_count: 0,
})
end
end
Expand Down