Query-Frontend: Add middleware to drop headers

[joe-elliott]

What this PR does:
Adds a new middleware to the front of all http pipelines that drops all headers except for a configurable list. This is done primarily as a performance improvement to prevent the unnecessary copying and recopying of headers to all sub jobs.

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[javiermolinar]

I think this is going to break Tempo since we are dropping the X-Scope-OrgID header. Maybe we should initialize the AllowedHearders config and hardcode the ones that Tempo needs

[joe-elliott]

I think this is going to break Tempo since we are dropping the X-Scope-OrgID header. Maybe we should initialize the AllowedHearders config and hardcode the ones that Tempo needs

prepareRequestForQueriers adds all required headers for the queriers including X-Scope-OrgID. This is called on all pipelines by the job sharder. the one exception to this is the metrics summary pipeline which doesn't do job sharding. i removed the header strip ware from that pipeline

[javiermolinar]

I think this is going to break Tempo since we are dropping the X-Scope-OrgID header. Maybe we should initialize the AllowedHearders config and hardcode the ones that Tempo needs

prepareRequestForQueriers adds all required headers for the queriers including X-Scope-OrgID. This is called on all pipelines by the job sharder. the one exception to this is the metrics summary pipeline which doesn't do job sharding. i removed the header strip ware from that pipeline

Right, I see that we inject the tenant in the auth middleware which happens before stripping the headers