Not able to read new log lines. but reading when vector restarted in Ubuntu

[kbitra]

I'm running vector in ubuntu. I have setup vector to read logs from files.

My setup is working and when new file added to the directory it's reading the data from the file. but when I add new lines to the existing files it's not reading.

When I stop and start vector. the new lines I added are being read. are there any settings I need to do in order to read the new lines.

My sample vector.
`
data_dir = "/home/user/vector/data"

#####CHANGETHIS Logs ########
[sources.test_log]
type = "file"
include = ["/home/user/logs/logtest*.log"]
read_from = "end"
`
I even tried adding the line_delimiter but still the same.

thanks

[jszwedko]

Hmm, the default behavior will be to read new lines. Can you share the debug log output of Vector? Also which version?

[kbitra]

Thanks for the reply.. attached is the debug log
vectordebuglog.txt

Version of Vector 0.31

Below is my toml file

`data_dir = "/home/user/vector/data"

[sources.simple]
type = "file"
include = ["/home/user/logs/simple.log"]
read_from = "beginning"
#read_from = "end"

[sinks.print]
type = "console"
inputs = ["simple"]
encoding.codec = "text"
`

[jszwedko]

Thanks for sharing @kbitra !

Unfortunately I'm not able to reproduce this behavior. I ran the following configuration:

data_dir = "/tmp/vector/"
[sources.source0]
type = "file"
include = ["/tmp/logs/simple.log"]
read_from = "beginning"

[sinks.sink0]
type = "console"
inputs = ["source0"]
encoding.codec = "json"

I then ran:

/tmp/logs on ☁️
❯ echo hello > simple.log

/tmp/logs on ☁️
❯ echo hello2 > simple.log

in a separate shell.

I saw both lines picked up by Vector:

2023-08-22T20:02:44.091954Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=info,rdkafka=info,buffers=info,lapin=info,kube=info"
2023-08-22T20:02:44.093535Z  INFO vector::app: Loading configs. paths=["/tmp/tmp.toml"]
2023-08-22T20:02:44.096978Z  INFO vector::topology::running: Running healthchecks.
2023-08-22T20:02:44.098113Z  INFO vector::topology::builder: Healthcheck passed.
2023-08-22T20:02:44.098391Z  INFO vector: Vector has started. debug="false" version="0.31.0" arch="aarch64" revision=""
2023-08-22T20:02:44.098983Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2023-08-22T20:02:44.098927Z  INFO source{component_kind="source" component_id=source0 component_type=file component_name=source0}: vector::sources::file: Starting file server. include=["/tmp/logs/simple.log"] exclude=[]
2023-08-22T20:02:44.101533Z  INFO source{component_kind="source" component_id=source0 component_type=file component_name=source0}:file_server: file_source::checkpointer: Attempting to read legacy checkpoint files.
2023-08-22T20:02:52.333711Z  WARN source{component_kind="source" component_id=source0 component_type=file component_name=source0}:file_server: vector::internal_events::file::source: Currently ignoring file too small to fingerprint. file=/tmp/logs/simple.log
2023-08-22T20:02:58.487583Z  INFO source{component_kind="source" component_id=source0 component_type=file component_name=source0}:file_server: vector::internal_events::file::source: Found new file to watch. file=/tmp/logs/simple.log
{"file":"/tmp/logs/simple.log","host":"COMP-J4C4P27K9Q","message":"hello","source_type":"file","timestamp":"2023-08-22T20:02:58.488191Z"}
2023-08-22T20:03:06.704499Z  INFO source{component_kind="source" component_id=source0 component_type=file component_name=source0}:file_server: vector::internal_events::file::source: Found new file to watch. file=/tmp/logs/simple.log
2023-08-22T20:03:06.704586Z  INFO source{component_kind="source" component_id=source0 component_type=file component_name=source0}:file_server: vector::internal_events::file::source: Stopped watching file. file=/tmp/logs/simple.log
{"file":"/tmp/logs/simple.log","host":"COMP-J4C4P27K9Q","message":"hello2","source_type":"file","timestamp":"2023-08-22T20:03:06.704823Z"}

[kbitra]

At First thank for taking time to reproduce..I think I’m running into Ubuntu clock issue.. the way vector detects the change to the file..  I’m just stating here if someone has encountered that..  I have syncd the clock etc., I think that could be the issue.. and vector thinking it’s not able to recognize the change..Sent from my iPhoneOn Aug 22, 2023, at 3:05 PM, Jesse Szwedko ***@***.***> wrote: Thanks for sharing @kbitra ! Unfortunately I'm not able to reproduce this behavior. I ran the following configuration: data_dir = "/tmp/vector/" [sources.source0] type = "file" include = ["/tmp/logs/simple.log"] read_from = "beginning" [sinks.sink0] type = "console" inputs = ["source0"] encoding.codec = "json" I then ran: /tmp/logs on ☁️ ❯ echo hello > simple.log /tmp/logs on ☁️ ❯ echo hello2 > simple.log in a separate shell. I saw both lines picked up by Vector: 2023-08-22T20:02:44.091954Z INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=info,rdkafka=info,buffers=info,lapin=info,kube=info" 2023-08-22T20:02:44.093535Z INFO vector::app: Loading configs. paths=["/tmp/tmp.toml"] 2023-08-22T20:02:44.096978Z INFO vector::topology::running: Running healthchecks. 2023-08-22T20:02:44.098113Z INFO vector::topology::builder: Healthcheck passed. 2023-08-22T20:02:44.098391Z INFO vector: Vector has started. debug="false" version="0.31.0" arch="aarch64" revision="" 2023-08-22T20:02:44.098983Z INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`. 2023-08-22T20:02:44.098927Z INFO source{component_kind="source" component_id=source0 component_type=file component_name=source0}: vector::sources::file: Starting file server. include=["/tmp/logs/simple.log"] exclude=[] 2023-08-22T20:02:44.101533Z INFO source{component_kind="source" component_id=source0 component_type=file component_name=source0}:file_server: file_source::checkpointer: Attempting to read legacy checkpoint files. 2023-08-22T20:02:52.333711Z WARN source{component_kind="source" component_id=source0 component_type=file component_name=source0}:file_server: vector::internal_events::file::source: Currently ignoring file too small to fingerprint. file=/tmp/logs/simple.log 2023-08-22T20:02:58.487583Z INFO source{component_kind="source" component_id=source0 component_type=file component_name=source0}:file_server: vector::internal_events::file::source: Found new file to watch. file=/tmp/logs/simple.log {"file":"/tmp/logs/simple.log","host":"COMP-J4C4P27K9Q","message":"hello","source_type":"file","timestamp":"2023-08-22T20:02:58.488191Z"} 2023-08-22T20:03:06.704499Z INFO source{component_kind="source" component_id=source0 component_type=file component_name=source0}:file_server: vector::internal_events::file::source: Found new file to watch. file=/tmp/logs/simple.log 2023-08-22T20:03:06.704586Z INFO source{component_kind="source" component_id=source0 component_type=file component_name=source0}:file_server: vector::internal_events::file::source: Stopped watching file. file=/tmp/logs/simple.log {"file":"/tmp/logs/simple.log","host":"COMP-J4C4P27K9Q","message":"","source_type":"file","timestamp":"2023-08-22T20:03:06.704791Z"} {"file":"/tmp/logs/simple.log","host":"COMP-J4C4P27K9Q","message":"hello2","source_type":"file","timestamp":"2023-08-22T20:03:06.704823Z"} —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>