HTTP config provider fails since 0.38 #21873

rmrustem · 2024-11-23T23:15:27Z

A note for the community

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Though not having been documented, obtaining the pipeline config on a HTTP endpoint introduced in #7166 worked perfectly until 0.37.1, but started to fails since 0.38.

Has this feature been deprecated and not offiicially supported or could it be fixed? Thank you!

Configuration

[provider]
  type = "http"
  url = "http://localhost/vector.toml"

# http://localhost/vector.toml
[sources.demo]
  type = "demo_logs"
  format = "syslog"
  interval = 1

[sinks.print]
  type = "console"
  encoding = { codec = "json", json = {pretty = true }}
  inputs = ["demo"]



### Version

vector 0.38.0 (x86_64-unknown-linux-gnu ea0ec6f 2024-05-07 14:34:39.794027186)

### Debug Output

```text
2024-11-23T23:07:54.422719Z DEBUG vector::app: Internal log rate limit configured. internal_log_rate_secs=10
2024-11-23T23:07:54.422783Z  INFO vector::app: Log level is enabled. level="trace"
2024-11-23T23:07:54.422928Z DEBUG vector::app: messaged="Building runtime." worker_threads=6
2024-11-23T23:07:54.423099Z TRACE mio::poll: registering event source with poller: token=Token(1), interests=READABLE    
2024-11-23T23:07:54.424491Z  INFO vector::app: Loading configs. paths=["web.toml"]
2024-11-23T23:07:54.425183Z DEBUG vector::config::loading: No secret placeholder found, skipping secret resolution.
2024-11-23T23:07:54.432455Z DEBUG vector_core::tls::settings: Fetching system root certs.
2024-11-23T23:07:54.438298Z DEBUG vector_core::tls::settings: Fetching system root certs.
2024-11-23T23:07:54.444450Z  INFO vector::providers::http: Attempting to retrieve configuration. url="http://localhost/vector.toml"
2024-11-23T23:07:54.444512Z DEBUG http: vector::internal_events::http_client: Sending HTTP request. uri=http://localhost/vector.toml method=GET version=HTTP/1.1 headers={"user-agent": "Vector/0.38.0 (x86_64-unknown-linux-gnu ea0ec6f 2024-05-07 14:34:39.794027186)", "accept-encoding": "identity"} body=[empty]
2024-11-23T23:07:54.444573Z TRACE http: hyper::client::pool: checkout waiting for idle connection: ("http", localhost)
2024-11-23T23:07:54.444617Z TRACE http: hyper::client::connect::http: Http::connect; scheme=Some("http"), host=Some("localhost"), port=None
2024-11-23T23:07:54.444997Z DEBUG hyper::client::connect::dns: resolving host="localhost"
2024-11-23T23:07:54.543314Z DEBUG http: hyper::client::connect::http: connecting to 127.0.0.1:80
2024-11-23T23:07:54.543459Z TRACE http: mio::poll: registering event source with poller: token=Token(139989083612032), interests=READABLE | WRITABLE    
2024-11-23T23:07:54.543576Z DEBUG http: hyper::client::connect::http: connected to 127.0.0.1:80
2024-11-23T23:07:54.545001Z TRACE http: hyper::client::conn: client handshake Http1
2024-11-23T23:07:54.545029Z TRACE http: hyper::client::client: handshake complete, spawning background dispatcher task
2024-11-23T23:07:54.545177Z TRACE http: hyper::client::pool: checkout dropped for ("http", localhost)
2024-11-23T23:07:54.545638Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Busy }
2024-11-23T23:07:54.545740Z TRACE encode_headers: hyper::proto::h1::role: Client::encode method=GET, body=None
2024-11-23T23:07:54.545840Z DEBUG hyper::proto::h1::io: flushed 165 bytes
2024-11-23T23:07:54.545872Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: KeepAlive, keep_alive: Busy }
2024-11-23T23:07:54.546029Z TRACE hyper::proto::h1::conn: Conn::read_head
2024-11-23T23:07:54.546636Z TRACE hyper::proto::h1::io: received 435 bytes
2024-11-23T23:07:54.546671Z TRACE parse_headers: hyper::proto::h1::role: Response.parse bytes=435
2024-11-23T23:07:54.546693Z TRACE parse_headers: hyper::proto::h1::role: Response.parse Complete(253)
2024-11-23T23:07:54.546721Z DEBUG hyper::proto::h1::io: parsed 8 headers
2024-11-23T23:07:54.546731Z DEBUG hyper::proto::h1::conn: incoming body is content-length (182 bytes)
2024-11-23T23:07:54.546804Z TRACE hyper::proto::h1::decode: decode; state=Length(182)
2024-11-23T23:07:54.546817Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-11-23T23:07:54.546836Z TRACE hyper::proto::h1::conn: maybe_notify; read_from_io blocked
2024-11-23T23:07:54.546850Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
2024-11-23T23:07:54.546845Z DEBUG http: vector::internal_events::http_client: HTTP response. status=200 OK version=HTTP/1.1 headers={"server": "nginx/1.18.0 (Ubuntu)", "date": "Sat, 23 Nov 2024 23:07:54 GMT", "content-type": "application/yaml", "content-length": "182", "last-modified": "Sat, 23 Nov 2024 14:56:05 GMT", "connection": "keep-alive", "etag": "\"6741ed05-b6\"", "accept-ranges": "bytes"} body=[182 bytes]
2024-11-23T23:07:54.546861Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
2024-11-23T23:07:54.546884Z  INFO vector::providers::http: Response received. url="http://localhost/vector.toml"
2024-11-23T23:07:54.546918Z TRACE hyper::client::pool: put; add idle connection for ("http", localhost)
2024-11-23T23:07:54.546959Z DEBUG hyper::client::pool: pooling idle connection for ("http", localhost)
2024-11-23T23:07:54.547038Z TRACE hyper::proto::h1::dispatch: client tx closed
2024-11-23T23:07:54.547056Z TRACE hyper::proto::h1::conn: State::close_read()
2024-11-23T23:07:54.547072Z TRACE hyper::proto::h1::conn: State::close_write()
2024-11-23T23:07:54.547120Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Closed, writing: Closed, keep_alive: Disabled }
2024-11-23T23:07:54.547174Z TRACE hyper::proto::h1::conn: shut down IO complete
2024-11-23T23:07:54.547196Z TRACE mio::poll: deregistering event source from poller    
2024-11-23T23:07:54.547280Z TRACE hyper::client::pool: pool closed, canceling idle interval
2024-11-23T23:07:54.547953Z ERROR vector::cli: Configuration error. error=TOML parse error at line 1, column 1
  |
1 | [sources.demo]
  | ^
wanted exactly 1 element, more than 1 element

Example Data

No response

Additional Context

No response

References

No response

The text was updated successfully, but these errors were encountered:

pront · 2024-11-27T19:35:49Z

Hi @rmrustem,

Thank you for creating this issue! Unfortunately, we currently don't have the capacity to work on this feature ourselves. However, we don't plan to remove it. If the community is motivated to work on it, we would be happy to review any contributions.

jszwedko · 2024-12-02T20:48:22Z

To add some more context, this feature was never officially released. It has seen some usage by astute users that came across its existence but testing around the feature is probably weaker than other Vector features.

jszwedko · 2024-12-02T21:43:42Z

I bisected this regression down to 8ed9ec2.

@jszwedko

Problem: The HTTP config provider has been failing to parse the provided TOML from its endpoint when polled. My best guess is that including the `?` on the end of the http provider load results in the TOML parser subtly exploding due to the aliased return type (`BuildResult`). It appears that this was broken in `8ed9ec24c1eba0b2191d7c1f24ec2a7540b5bebf`. See vectordotdev#21873 for the initial issue report. Solution: Correctly propagate the `ConfigBuilder` by not using the `?` operator. This appears to allow the TOML parser to correctly pick up on the return type. ----- Thanks to @jszwedko for the bisect, saved a ton of pain for me. closes vectordotdev#21873

@jszwedko

Problem: The HTTP config provider has been failing to parse the provided TOML from its endpoint when polled. My best guess is that including the `?` on the end of the http provider load results in the TOML parser subtly exploding due to the aliased return type (`BuildResult`). It appears that this was broken in `8ed9ec24c1eba0b2191d7c1f24ec2a7540b5bebf`. See vectordotdev#21873 for the initial issue report. Solution: Correctly propagate the `ConfigBuilder` by not using the `?` operator. This appears to allow the TOML parser to correctly pick up on the return type. ----- Thanks to @jszwedko for the bisect, saved a ton of pain for me. closes vectordotdev#21873

@jszwedko

Problem: The HTTP config provider has been failing to parse the provided TOML from its endpoint when polled. My best guess is that including the `?` on the end of the http provider load results in the TOML parser subtly exploding due to the aliased return type (`BuildResult`). It appears that this was broken in `8ed9ec24c1eba0b2191d7c1f24ec2a7540b5bebf`. See vectordotdev#21873 for the initial issue report. Solution: Correctly propagate the `ConfigBuilder` by not using the `?` operator. This appears to allow the TOML parser to correctly pick up on the return type. ----- Thanks to @jszwedko for the bisect, saved a ton of pain for me. closes vectordotdev#21873

@jszwedko

Problem: The HTTP config provider has been failing to parse the provided TOML from its endpoint when polled. My best guess is that including the `?` on the end of the http provider load results in the TOML parser subtly exploding due to the aliased return type (`BuildResult`). It appears that this was broken in `8ed9ec24c1eba0b2191d7c1f24ec2a7540b5bebf`. See vectordotdev#21873 for the initial issue report. Solution: Correctly propagate the `ConfigBuilder` by not using the `?` operator. This appears to allow the TOML parser to correctly pick up on the return type. ----- Thanks to @jszwedko for the bisect, saved a ton of pain for me. closes vectordotdev#21873

@jszwedko

#21982) Problem: The HTTP config provider has been failing to parse the provided TOML from its endpoint when polled. My best guess is that including the `?` on the end of the http provider load results in the TOML parser subtly exploding due to the aliased return type (`BuildResult`). It appears that this was broken in `8ed9ec24c1eba0b2191d7c1f24ec2a7540b5bebf`. See #21873 for the initial issue report. Solution: Correctly propagate the `ConfigBuilder` by not using the `?` operator. This appears to allow the TOML parser to correctly pick up on the return type. ----- Thanks to @jszwedko for the bisect, saved a ton of pain for me. closes #21873

rmrustem added the type: bug A code related bug. label Nov 23, 2024

jszwedko added the domain: config Anything related to configuring Vector label Dec 2, 2024

PriceHiller mentioned this issue Dec 8, 2024

fix(config): correctly propagate http config provider Result to caller #21982

Merged

10 tasks

pront closed this as completed in #21982 Dec 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HTTP config provider fails since 0.38 #21873

HTTP config provider fails since 0.38 #21873

rmrustem commented Nov 23, 2024 •

edited

Loading

pront commented Nov 27, 2024

jszwedko commented Dec 2, 2024

jszwedko commented Dec 2, 2024

HTTP config provider fails since 0.38 #21873

HTTP config provider fails since 0.38 #21873

Comments

rmrustem commented Nov 23, 2024 • edited Loading

A note for the community

Problem

Configuration

Example Data

Additional Context

References

pront commented Nov 27, 2024

jszwedko commented Dec 2, 2024

jszwedko commented Dec 2, 2024

rmrustem commented Nov 23, 2024 •

edited

Loading