Move to full asynchronous runner

[augustuswm]

This issue is a tracking issue for a rework of the job runner model that cio-bot uses.

Motivation

CIO does a good job of providing a readable (in code) encapsulation of the business rules and processes that run many day to day operations that are otherwise often spread across multiple departments and people. Currently it struggles though in reporting back what work it has done and why. Parts of the execution (specifically cron jobs) collect logs, but they are unstructured and lose their level designation when globally integrated. Webhook handlers track separate data to global logs storage, GitHub commit comments, and GitHub Check Runs. Additionally all handlers send some portion of logs (warnings and errors) to Sentry.

As such in the current state, asking "What did handler X do?" or "Why did handler Y do Z?" requires tracing the cio codebase to understand how that handler reports its data. While there are some patterns that can be learned as a starting point, we can make this significantly more effective.

To summarize, we are specifically interested in improving "how" cio does the work it does, as opposed to "what" work cio does.

Goals

The key goals that we want to hit during this implementation are:

• Retain all existing functionality of current cio handlers
• Collect logs and tracing metadata for all handlers in a global log store
• Store payloads of accepted incoming events
• Track handlers that have been run against a given event
• Update webhook handlers to immediately respond to requests, and execute work asynchronously
• Allow redelivery of events
• Allow re-running of handlers
• Generalize cron and webhook handlers in to a single execution model
• Merge cron and event handlers so that their core functionality is the same