enhancement(http source): unifying http query parameters #22242

sainad2222 · 2025-01-18T19:43:50Z

Summary

Unifying http query parameters while maintaining backward compatibility. Refer to this for approach

Change Type

Bug fix
New feature
Non-functional (chore, refactoring, docs)
Performance

Is this a breaking change?

Yes
No

How did you test this PR?

sources:
  prom_source:
    endpoints:
      - "http://localhost:9090/federate"
    honor_labels: true
    scrape_interval_secs: 60
    scrape_timeout_secs: 45
    query:
      match[]:
        - '{job="node1"}'
    tls:
      verify_certificate: false
      verify_hostname: false
    type: prometheus_scrape

sinks:
  output:
    type: "console"
    inputs: ["prom_source"]
    target: "stdout"
    encoding:
      codec: "json"

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the "no-changelog" label to this PR.

Checklist

Please read our Vector contributor resources.
- make check-all is a good command to run locally. This check is
  defined here. Some of these
  checks might not be relevant to your PR. For Rust changes, at the very least you should run:
  - cargo fmt --all
  - cargo clippy --workspace --all-targets -- -D warnings
  - cargo nextest run --workspace (alternatively, you can run cargo test --all)
If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
run dd-rust-license-tool write to regenerate the license inventory and commit the changes (if any). More details here.

References

Closes: Unify handling of query for HTTP-based components that support it #12073

kayayarai

No docs review required

sainad2222 · 2025-01-22T17:52:52Z

@jszwedko Gentle ping :)

jszwedko

Nice! Thanks for this improvement @sainad2222 ! I think the code looks good. It is a bit unfortunate that the docs don't clearly show that query can be a key/value or key/values. I expect users may end up confused. If you feel up to it, you could try to dig into the docs generation to see if you can adjust it, but I don't think that needs to block this since the current docs also don't really clearly show how the query option is used.

I left a couple of other minor comments below.

jszwedko · 2025-01-23T23:12:43Z

src/sinks/elasticsearch/config.rs

@@ -175,7 +175,7 @@ pub struct ElasticsearchConfig {
    #[configurable(metadata(docs::advanced))]
    #[configurable(metadata(docs::additional_props_description = "A query string parameter."))]
    #[configurable(metadata(docs::examples = "query_examples()"))]


It seems like these examples aren't rendering properly on the docs, but that seems to be an existing issue so we don't need to block this.

However, could we add to the examples an example of specifying a single value? I think all existing examples in query_examples() for all of the sources only show specifying an array of values.

As an alternative, you can hand write a small "how it works" section on query params

Hi,
@jszwedko This particular example has single value(for elasticsearch sink) only. Did you mean for http_client source? If Yes, I made changes for that now
@pront Where can I write this "how it works" for query params. Any existing reference I can follow? Will it go in .rs files or .cue files?

You can find an example here: https://github.com/vectordotdev/vector/blob/master/website/cue/reference/components/transforms/exclusive_route.cue#L51. Let me know if there are more questions.

Thanks this helps. Just to clarify, right now changes are in 3 places

elasticsearch(single params only)

prometheus_scrape(multiple params or single param but query key name should include [])

http_client(generic, so either single param or multiple params)

So Do I need to create how_it_works section individually for each of their cue files with these details along with examples?

Ah good point. It seems we are hitting a lot of doc generation limitations here. I think we should copy paste it in all affected components in this case.

jszwedko · 2025-01-23T23:16:07Z

website/cue/reference/components/sources/base/http_client.cue

@@ -515,7 +515,7 @@ base: components: sources: http_client: configuration: {
 			options: "*": {
 				description: "A query string parameter and it's value(s)."
 				required:    true
-				type: array: items: type: string: {}


Technically this should indicate that either a string or [string] can be used, but I'm not sure the current cue schema supports this (or if the code that generates the cue from the struct definitions does) 🤔

pront

👍 from me, would be great to get another review from @jszwedko for UX

jszwedko

Thanks for adding the examples!

sainad2222 · 2025-02-03T08:48:27Z

@pront Need some help in debugging nats integration test failure. I see two types of errors

publish_and_check failed, expected BuildError::Connect, got: Err(Config { source: CredentialsFileError { source: Custom { kind: InvalidData, error: Error { kind: ChecksumFailure, description: Some("Checksum mismatch") } } } })

and

+ datadog-ci junit upload --service vector target/nextest/default/junit.xml
./scripts/upload-test-results.sh: line 22: datadog-ci: command not found
+ echo 'Failed to upload results'

Second one might be a side effect of first. But I am not sure why integration test may panic because of my changes

…s_unifying_12073

pront · 2025-02-03T14:40:28Z

@pront Need some help in debugging nats integration test failure. I see two types of errors
publish_and_check failed, expected BuildError::Connect, got: Err(Config { source: CredentialsFileError { source: Custom { kind: InvalidData, error: Error { kind: ChecksumFailure, description: Some("Checksum mismatch") } } } })
and
+ datadog-ci junit upload --service vector target/nextest/default/junit.xml
./scripts/upload-test-results.sh: line 22: datadog-ci: command not found
+ echo 'Failed to upload results'
Second one might be a side effect of first. But I am not sure why integration test may panic because of my changes

Hmm, not sure. I rebased on origin/master, if it happens again I will take a closer look.

sainad2222 · 2025-02-03T16:08:22Z

@pront Can you trigger workflows too?

sainad2222 · 2025-02-03T18:05:55Z

Still failing 🤔 . Can't find any recent PRs that has integration tests run for nats too

pront · 2025-02-03T18:32:17Z

Still failing 🤔 . Can't find any recent PRs that has integration tests run for nats too

This is probably an unrelated failure. I don't have time right now but I plan to take a look soon. Unless you beat me to it ;)

pront · 2025-02-03T22:31:43Z

Still failing 🤔 . Can't find any recent PRs that has integration tests run for nats too

This is probably an unrelated failure. I don't have time right now but I plan to take a look soon. Unless you beat me to it ;)

Made a test PR: #22359 (review)

Kicked off a run: https://github.com/vectordotdev/vector/actions/runs/13124105116/job/36616694045?pr=22359

sainad2222 · 2025-02-04T07:57:41Z

What's the correct way to run integration tests locally? I tried doing docker-compose -f nats/compose.yaml up and then cargo test --features "nats-integration-tests" sinks::nats::integration_tests::nats_tls_valid but it seems to have tls error this way. Github actions looks to be using vector test runner so couldn't figure out the steps from the logs

Exact error logs

tests logs

thread 'sinks::nats::integration_tests::nats_tls_valid' panicked at src/sinks/nats/integration_tests.rs:313:5:
publish_and_check failed, expected Ok(()), got: Err(Connect { source: Error { kind: Io, source: Some(Custom { kind: InvalidData, error: InvalidMessage(InvalidContentType) }) } })

nats server logs

[nats]                 | [1] 2025/02/04 07:55:48.757058 [DBG] 10.89.3.52:55782 - cid:9 - Client connection created
[nats]                 | [1] 2025/02/04 07:55:48.764665 [TRC] 10.89.3.52:55782 - cid:9 - ->> [-ERR Unknown Protocol Operation]
[nats]                 | [1] 2025/02/04 07:55:48.765445 [ERR] 10.89.3.52:55782 - cid:9 - Client parser ERROR, state=0, i=0: proto='"\x16\x03\x01\x00\xd5\x01\x00\x00\xd1\x03\x03\x11Ҏ\x80<\n\xbe\x81\x9c\x1387C\xe1\x9b1\xeaC\x98\xb3\xf3"...'
[nats]                 | [1] 2025/02/04 07:55:48.765492 [DBG] 10.89.3.52:55782 - cid:9 - Client connection closed: Protocol Violation
[nats]                 | [1] 2025/02/04 07:55:48.767593 [DBG] 10.89.3.52:55784 - cid:10 - Client connection created
[nats]                 | [1] 2025/02/04 07:55:48.769498 [TRC] 10.89.3.52:55784 - cid:10 - ->> [-ERR Unknown Protocol Operation]
[nats]                 | [1] 2025/02/04 07:55:48.769557 [ERR] 10.89.3.52:55784 - cid:10 - Client parser ERROR, state=0, i=0: proto='"\x16\x03\x01\x00\xd5\x01\x00\x00\xd1\x03\x03\x82\xa4rݿ\x9b*\xcdE\x98\x8fo\xc7ր3\x1e\t*\xfa\v"...'
[nats]                 | [1] 2025/02/04 07:55:48.769561 [DBG] 10.89.3.52:55784 - cid:10 - Client connection closed: Protocol Violation

Also might not be related but there is a recent upgrade of async-nats library and integration tests didn't run for that PR too. They got skipped

pront · 2025-02-04T14:55:41Z

#22359 (review)

You should be able to run with:

scripts/ci-int-e2e-test.sh int nats

I noticed some weirdness, on my run as well. The datadog-e2e-metrics test failed and the nats integration tests did not run (I suppose stopped early because of the first failure).

Running it here:
https://github.com/vectordotdev/vector/actions/runs/13138215406/job/36658665492?pr=22359

pront · 2025-02-05T17:13:18Z

You can do a git merge origin master and push to pick up the fix. Apologies for the friction here but it exposed an interesting bug.

…nifying_12073

sainad2222 · 2025-02-05T18:49:37Z

splunk tests are taking ~1h. Is it expected to take this time?

pront · 2025-02-05T19:15:01Z

splunk tests are taking ~1h. Is it expected to take this time?

No, this looks like a regression. The test filter decided to run splunk tests because src/sources/util/http_client.rs was modified and it matches src/sources/util/**

enhancement(http): unifying http query parameters

bcbbb62

sainad2222 requested review from a team as code owners January 18, 2025 19:43

github-actions bot added domain: sources Anything related to the Vector's sources domain: sinks Anything related to the Vector's sinks domain: external docs Anything related to Vector's external, public documentation labels Jan 18, 2025

kayayarai reviewed Jan 20, 2025

View reviewed changes

jszwedko reviewed Jan 23, 2025

View reviewed changes

fix: update query_examples for http source

71c6193

sainad2222 requested review from pront and jszwedko January 27, 2025 18:25

docs: update cue files with how_it_works section

0145622

sainad2222 changed the title ~~enhancement(http): unifying http query parameters~~ enhancement(http_client): unifying http query parameters Jan 28, 2025

chore: add changelog

16787d3

sainad2222 changed the title ~~enhancement(http_client): unifying http query parameters~~ enhancement(http source): unifying http query parameters Jan 30, 2025

pront approved these changes Jan 31, 2025

View reviewed changes

jszwedko approved these changes Jan 31, 2025

View reviewed changes

pront enabled auto-merge February 1, 2025 20:31

Merge remote-tracking branch 'origin' into sainad2222/query_parameter…

9f72a35

…s_unifying_12073

pront mentioned this pull request Feb 3, 2025

chore(ci): make Integration Test Suite runnable on Actions #22356

Merged

10 tasks

This was referenced Feb 4, 2025

fix(ci): temporarily ignore nats integration tests #22364

Closed

fix(ci): revert to a previous async-nats version #22359

Merged

fix(ci): always run IT suite when in a merge group #22368

Merged

Merge branch 'vectordotdev:master' into sainad2222/query_parameters_u…

ca521f6

…nifying_12073

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enhancement(http source): unifying http query parameters #22242

enhancement(http source): unifying http query parameters #22242

sainad2222 commented Jan 18, 2025 •

edited by pront

Loading

kayayarai left a comment

sainad2222 commented Jan 22, 2025

jszwedko left a comment

jszwedko Jan 23, 2025

pront Jan 24, 2025

sainad2222 Jan 25, 2025

pront Jan 27, 2025

sainad2222 Jan 27, 2025

pront Jan 27, 2025

jszwedko Jan 23, 2025

pront left a comment

jszwedko left a comment

sainad2222 commented Feb 3, 2025

pront commented Feb 3, 2025

sainad2222 commented Feb 3, 2025

sainad2222 commented Feb 3, 2025

pront commented Feb 3, 2025

pront commented Feb 3, 2025

sainad2222 commented Feb 4, 2025

pront commented Feb 4, 2025

pront commented Feb 5, 2025

sainad2222 commented Feb 5, 2025

pront commented Feb 5, 2025 •

edited

Loading

enhancement(http source): unifying http query parameters #22242

Are you sure you want to change the base?

enhancement(http source): unifying http query parameters #22242

Conversation

sainad2222 commented Jan 18, 2025 • edited by pront Loading

Summary

Change Type

Is this a breaking change?

How did you test this PR?

Does this PR include user facing changes?

Checklist

References

kayayarai left a comment

Choose a reason for hiding this comment

sainad2222 commented Jan 22, 2025

jszwedko left a comment

Choose a reason for hiding this comment

jszwedko Jan 23, 2025

Choose a reason for hiding this comment

pront Jan 24, 2025

Choose a reason for hiding this comment

sainad2222 Jan 25, 2025

Choose a reason for hiding this comment

pront Jan 27, 2025

Choose a reason for hiding this comment

sainad2222 Jan 27, 2025

Choose a reason for hiding this comment

pront Jan 27, 2025

Choose a reason for hiding this comment

jszwedko Jan 23, 2025

Choose a reason for hiding this comment

pront left a comment

Choose a reason for hiding this comment

jszwedko left a comment

Choose a reason for hiding this comment

sainad2222 commented Feb 3, 2025

pront commented Feb 3, 2025

sainad2222 commented Feb 3, 2025

sainad2222 commented Feb 3, 2025

pront commented Feb 3, 2025

pront commented Feb 3, 2025

sainad2222 commented Feb 4, 2025

Exact error logs

tests logs

nats server logs

pront commented Feb 4, 2025

pront commented Feb 5, 2025

sainad2222 commented Feb 5, 2025

pront commented Feb 5, 2025 • edited Loading

sainad2222 commented Jan 18, 2025 •

edited by pront

Loading

pront commented Feb 5, 2025 •

edited

Loading