I'm not convinced this PR is addressing the irregular sampling behaviour of the compare() function.

Running the query range e2e test, I get very similar results to before, only with exemplars returned in a different series.

IMO, the first step should be extending that test or creating a new one to be able to verify the behaviour we're looking for: evenly distributed exemplars across all series.

Previously only the series labeled with baseline_total and selection_total had exemplars. Now all can have exemplars. This doesn't mean that all the series are going to have at least one exemplar due to the way the bucket works. Sampling by interval only, fixing that will require much more work.

I'll work on improving the test

I'm not convinced this PR is addressing the irregular sampling behaviour of the compare() function.
Running the query range e2e test, I get very similar results to before, only with exemplars returned in a different series.
IMO, the first step should be extending that test or creating a new one to be able to verify the behaviour we're looking for: evenly distributed exemplars across all series.

Previously only the series labeled with baseline_total and selection_total had exemplars. Now all can have exemplars. This doesn't mean that all the series are going to have at least one exemplar due to the way the bucket works. Sampling by interval only, fixing that will require much more work.

I'll work on improving the test

Right, these changes add exemplars to more series, but they don't address the core issues described in #4284—which are that sampling is not working as expected and it's returning too few exemplars in general. It's a frequent occurence that in a compare() query most series (_totals or not) don't have exemplars, even though there are traces that match those series.

Also, I'm not sure that adding exemplars to baseline and selection series is an improvement over baseline_total and selection_total, as the latter are supersets of the former.