Speedup collection of results from ingesters in the querier (#4100)

* Add Generic Collector for collection of things

* push collection down to calls to ingester

* fixup generic collector

* cleanup comments and TODOs

* Update CHANGELOG.md

* match span name with func name

* inline the collection of results in SearchRecent

* remove break goto

* make ScopedDistinctString and DistinctString safe + test

CHANGELOG.md

@@ -1,5 +1,6 @@
 ## main / unreleased
 
+* [ENHANCEMENT] Speedup collection of results from ingesters in the querier [#4100](https://github.com/grafana/tempo/pull/4100) (@electron0zero)
 * [ENHANCEMENT] Speedup DistinctValue collector and exit early for ingesters [#4104](https://github.com/grafana/tempo/pull/4104) (@electron0zero)
 * [ENHANCEMENT] Add disk caching in ingester SearchTagValuesV2 for completed blocks [#4069](https://github.com/grafana/tempo/pull/4069) (@electron0zero)
 * [BUGFIX] Replace hedged requests roundtrips total with a counter. [#4063](https://github.com/grafana/tempo/pull/4063) [#4078](https://github.com/grafana/tempo/pull/4078) (@galalen)

modules/querier/http.go

@@ -290,7 +290,7 @@ func (q *Querier) SearchTagValuesV2Handler(w http.ResponseWriter, r *http.Reques
 	ctx, cancel := context.WithDeadline(r.Context(), time.Now().Add(q.cfg.Search.QueryTimeout))
 	defer cancel()
 
-	ctx, span := tracer.Start(ctx, "Querier.SearchTagValuesHandler")
+	ctx, span := tracer.Start(ctx, "Querier.SearchTagValuesV2Handler")
 	defer span.End()
 
 	var resp *tempopb.SearchTagValuesV2Response