refactor(repository): improve usability of repository

[abelanger5]

Description

This PR consolidates the repository methods which had become fractured so they're all defined in a single package. It moves the buffers and the scheduler repositories into the same package as other repository methods. It also introduces 3 patterns for recent issues we've seen while developing complex features:

1. Transactions may need to be shared between methods that are defined on separate repositories. For example, to get transactional safety for workflow run versus step run state as part of Bugfix - stalled worklowruns  #1085, we need to run queries against step runs which were previous in the StepRunRepository. The solution in that PR (and elsewhere in the repo) was to pass StepRunRepository as a field dependency of WorkflowRunRepository.

   There is now a sharedRepository which can be used to share underlying methods and transactions. As an example, I've modified the DeferredStepRunEvent (previously called from WorkflowRunRepository) to:

     func (s *sharedRepository) deferredStepRunEvent(
       tenantId string,
       opts repository.CreateStepRunEventOpts,
   ) {
       // fire-and-forget for events
       err := s.bulkEventBuffer.FireForget(tenantId, &opts)
   
       if err != nil {
   	    s.l.Error().Err(err).Msg("could not buffer event")
       }
   }

   Each repository should embed *sharedRepository. This also makes initializing repositories a bit cleaner, as we don't need to pass the same validator, logger, or pgxpool around everywhere.

2. Similar to the above, buffers are now defined on the sharedRepository, which gives us a single place where we need to track buffers and ensure they're cleaned up. They can also be called across multiple separate repositories. I've made the additional change of changing the buffer signature from BuffItem, which previously returned a chan with a result or error, and modified the signature to the following:

   func FireForget(item T) error 
   
   func FireAndWait(ctx context.Context, item T) (*U, error)

   The reasoning here is that these are the only 2 patterns that have been used in the repository for buffering items, and dealing with multiple channels and context passed back to the caller is incredibly error-prone (I found a few instances of results not being awaited when they should have been). This also makes it much easier for a buffered write to respect a context timeout (we were previously setting a timer and adding an additional case for timeouts when awaiting results).

3. Callback patterns -- there are several instances where we've wanted to hook into a pub/sub message or other type of callback inside of repository methods ( this was a challenge with Feat skip states for workflows #1075 ), but we've been hesitant to do this because we don't want to pass a pub/sub dependency through to the repository.

   There's now an easy way to define callbacks on repository methods via the following:

   // In the core repository package, when defining the repository's interface:
   RepositoryMethod(ctx context.Context, p params, opts ...CallbackOptFunc[string]) error
   
   // In the method implementation
   func (w *repository) RepositoryMethod(ctx context.Context, p params, opts ...CallbackOptFunc[string]) error {
       // start tx
   
       repository.RunPreCommit(w.l, tenantId, workflowRunId, opts)
   
       // commit tx
   
       repository.RunPostCommit(w.l, tenantId, workflowRunId, opts)
   
       return nil
   }

   The callbacks are utilizing the same underlying Do method as previously, which is non-blocking and only logs errors. We may have to tweak signatures here if we want a failed pre-commit hook to fail the transaction.

Type of change

• Refactor (non-breaking changes to code which doesn't change any behaviour)

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
hatchet-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Dec 11, 2024 8:30pm

[reillyse]

Hard saying how easy/hard this will be without using the structure a bit but I like the direction.

[reillyse]

nice