Allow "Annotation" rules as a way to dry-run individual rules

[kentquirk]

Is your feature request related to a problem? Please describe.

Refinery's Dry Run mode is useful during the early stages of an implementation, but it doesn't help once things are in production -- switching to Dry Mode again is likely to be very expensive and might overwhelm infrastructure.

Describe the solution you'd like

It would be nice to have the ability to test a new rule without affecting existing telmetry, so here's a proposal:

In addition to setting a sample rate and Drop: true as rule actions, we could support something like Annotate: true.

This would cause Refinery to add a new field meta.annotation to the root span (or to every span?). It would be set to the name of the specific refinery rule but would otherwise not change your existing rule behavior (it would be treated like a rule that didn't match).

Because it behaves like a rule that doesn't match, it would have to be placed before any rule that does match. This also means that some things that would have been annotated will be dropped before reaching Honeycomb.

It might also be interesting to allow a way to treat it like a rule that does match, so that it behaves more like dry run but at the rule level.

Describe alternatives you've considered

As far as I know, we've never considered an idea like this before now.

Additional context

Based on a customer request (internal slack).