feat: Add global job timeout

[bgins]

Summary

This pull request makes the following changes:

• Add expired job cancellation to the solver control loop
  • Add JOB_TIMEOUT_SECONDS config
  • Add JobTimedOut terminal agreement state
  • Add cancelled job metrics
  • Add cancelled job traces
• Add Active job offer query filter
• Add Cancelled job offer query filter
  • Remove IncludeCancelled filter and integrate it into Cancelled

We have also made a couple of related changes and small cleanups:

• Rename deal stats metrics to job stats metrics
• Record job stat metrics for cancelled job offers that are not involved in a deal
• Clean up allowed resource providers between test runs. We missed it while rebasing feat: Add resource provider allowlist #481.
• Update comments in solver solve function
• Add BoolPointer language helper to more easily construct pointers to bools

The primary goal of this pull request is adding a global job timeout for jobs that are stuck, stale, or abandoned. The global timeout will be superseded by timeouts in module specs and job offers when we have implemented them.

We have also added new Active and Cancelled job offer query filters in support of the global timeout.

Task/Issue reference

Closes: #511

Test plan

Start the base services. Start the solver with the job offer timeout set to ten seconds:

./stack solver --job-timeout-seconds 10

Submit a job without a resource provider running. The job should fail and report a job timed out error message.

Start a resource provider and run another job. Again, the job should be cancelled, but this time in flight. The job creator should see the error message and the resource provider should repost their resource offer.

See the details below if you would like to confirm the activity through metrics and traces.

Details

Active and Cancelled job offer filter queries

We have added two new job offer filter queries.

When Active is true, only jobs in DealNegotiating, DealAgreed, or ResultsSubmitted states are included.

Query active job offers from the solver server with the active querstring param. In local development:

curl http://localhost:8081/api/v1/job_offers?active=true

The Cancelled query filter is a *bool where:

• nil includes all jobs
• true includes jobs in JobOfferCancelled and JobTimedOut states
• false includes jobs not in a JobOfferCancelled or JobTimedOut state

We removed the IncludeCancelled filter query which is now performed when Cancelled is nil.

Query jobs on the solver server using the cancelled querystring param. In local development:

curl http://localhost:8081/api/v1/job_offers                    # all jobs
curl http://localhost:8081/api/v1/job_offers?cancelled=true     # cancelled jobs   
curl http://localhost:8081/api/v1/job_offers?cancelled=false    # not cancelled jobs

JobTimedOut agreement state

We have added a new JobTimedOut agreement state to describe jobs that were cancelled because they expired.

We considered using the existing JobOfferCancelled state, but it is not coherent when we want to mark resource offers and deals with an expired state. In addition, our mechanism for reporting what happened to the job creator is limited, so JobTimedOut serves as a tag to help us log a good error message for the job creator.

When a job times out before a deal was made, the job offer is marked with the JobTimedOut state. When a deal has already been made, the job offer, resource offer, and deal are all marked with the JobTimedOut state.

If a resource provider attempts to upload files or report results for an expired job, the solver server rejects their post with an error reporting the job as timed out.

JOB_TIMEOUT_SECONDS config

We have added a new config to set the global timeout in seconds. The default value is ten minutes.

Set the job timeout with an environment variable of CLI option. For example, to expire jobs after 30 seconds:

./stack solver --job-timeout-seconds 30
JOB_TIMEOUT_SECONDS=30 ./stack solver

Observability

The deal metrics have been updated to reflect overall job metrics. We want to include metrics about job offers that were never in a deal, and we record these from jobs that expired or were cancelled.

The metrics are reported in an updated Solver Metrics dashboard:

The updated dashboard is available in this observability pull request: https://github.com/Lilypad-Tech/observability/pull/27

Metrics are not enabled by default in local development, but they can be turned on with:

export ENABLE_METRICS=true
export METRICS_URL="http://localhost:8500"
export METRICS_TOKEN="eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJhdXRob3JpemVkIjp0cnVlLCJ1c2VyIjoicmVzb3VyY2UtcHJvdmlkZXIifQ.n36M_ngwC4XPQ_pEkkWAnPiOinnx6-0VO1v_WgCTUEERD7b_p9KHCU6SY5bUdFh5UXRZHAhc1gfyc7rjAnmeDQ"

We have also included traces over our new cancelExpiredJobs solver controller function. Traces are also disabled by default in local development, but can be enabled with the --disable-telemetry=false CLI option.

The traces can be queried by searching TraceQL with the cancel_expired_jobs span name and a non-empty expired jobs span attribute:

{name="cancel_expired_jobs" && span.expired_job_offers!="[]"}

When a job is cancelled before a match, the trace will look like:

If a job is cancelled after a match, it will also report a deal ID:

Related issues or PRs

Observability dashboard PR: https://github.com/Lilypad-Tech/observability/pull/27

[narbs91]

LGTM!