TraceQL: Support select() on multiple scopes

[farcaller]

Why is this needed:

select() allows you to visualize the attributes of the span, making them easier to visually compare. However, it's not possible to select anything from parent scopes if the final selected span doesn't provide enough context.

What would you like to be added:

Consider the following traces:

<span name="hello" sender="alice">
  <span name="world" message="hello world" />
</span>

You can select all the world spans within helloes and add the column for message via { span:name = "hello" } >> { span:name = "world" } | select(span.message), however you can't add the sender of the parent span to vizualisation.

I don't know how the syntax for this should look like. My naive expectation is that { span:name = "hello" } | select(span.sender) >> { span:name = "world" } | select(span.message) would create a table with both sender and message columns.

Who is this feature for?

This is useful when you're analyzing the issue stretching across several nested spans, you narrowed it down to the specific parent-child spans and want to see the attributes in play.

[adrapereira]

Moved to https://github.com/grafana/tempo since this is a TraceQL feature request.

[mapno]

Hi @farcaller. Have you tried using union structural operators? It returns the matching spans of both selectors.

[joe-elliott]

I really like this idea. The ability to select (or even just reference) the attributes from other spans. The original TraceQL design actually included this with the concept of parent:

{ parent.foo = "bar" }

but we've struggled to build it into the language due to storage level issues. I would not give up hope tho! I want this as well, but it is definitely not a current focus of the team.

[farcaller]

Hi @farcaller. Have you tried using union structural operators? It returns the matching spans of both selectors.

I don't think that's semantically the same. The result is both spans and I can |select() the fields from both but now I'm human-parsing the output to match the lines. I'd rather have the machine do the job for me 😄

[joe-elliott]

I don't think that's semantically the same. The result is both spans and I can |select() the fields from both but now I'm human-parsing the output to match the lines. I'd rather have the machine do the job for me 😄

Agreed. I was going to suggest experimenting with using Grafana tables and transformations to link the two up, but while you can select(span:id) you can't select(span:parentID). we should add that to the language.

[joe-elliott]

It just occurred to me that you can do this with the nested set values. I know we're in super hacky territory here. I acknowledge this isn't a "real" solution but it would be interesting to see how well it would work. A parent span can be identified if nestedSetLeft = nestedSetParent

In this screenshot you can see the queried span has two children.

[mdisibio]

Agree it would be awesome to have more capabilities in this area of selecting/organizing output. The original idea of selecting on multiple spanset filters I'm thinking hits up against more limits in the language other than the parsing. Spanset operators returns the right-hand-side, so a query like {A} | select(...) >> {B} | select(...) would never return spans from {A} anyways. Therefore select()ing, even if it worked, wouldn't be available to display in the output.