Logs enhancements #3
Assignees
Comments
|
Thank you for the feedback. |
|
Here is an example of Zeek / passiverecon log, generated by Masscanned tests (tab separated): |
|
And here is an example of p0f (v3) output, also generated by Masscanned tests: |
|
I think some useful features towards logs enhancements would be: Support of "ingestible formats"As mentioned in the opening comment, using a more universal format like JSON or logfmt allows easy redirection to broad data aggregation platforms, and better later processing of these logs. e.g. when a key-value format like JSON is used, a process going through the logs doesn't have to think about how they're written, which column is the one with the value it needs, and that makes for clearer/shorter interaction. Format-agnostic logging, and format selectionThe codebase kind of already works like that / supports it if one adds his own Rust logger, but the tool as is does not. The idea being to have a CLI option to select the format in which to output logs, e.g. json, logfmt, tabular. Implementation-wise, this could be done by having the various Maybe through a struct with defaults (and Options for good measure?) so you can avoid telling the function things you don't have (e.g. IP addresses when logging Ethernet events) and let the log function handle it according to the format : just no key in something like JSON, an empty string in the tab-separated current format... The universal log function would then use the chosen format to print the data. Perfect consistency of all logs in regard to the formatIn its current state the arp logging is different from all the others (eth, ipv4, ipv6, icmpv4, icmpv6, tcp, udp). Configuring what to log from CLICurrently the logger just sets to true all protocols and logs everything. One could imagine an option to unset and say, don't print events from the ethernet protocol, or some verbs like Log application layer formats, in the same way as other protocolsCoupled with choosing what protocols to log, I think it would just be very neat. Include TCP/UDP ports in
|
|
Thank you for your input on the logging. Your ideas are interesting, and the logging part is definitely to be improved.
For any of these, it is obviously open to discussion, and also to PR if you'd like to contribute to the project. |
|
About the consistency question for application protocols, the issue I was highlighting was basically: "when you see a value at the nth line of output, how do you know what it corresponds to?" The issues with columns for this are:
What I then tried to highlight is that these issues actually all disappear once you use a key-value format like json or logfmt. So I'm unsure of the approach to take when using a column format like the current one, and how much it can be solved in that context without any form of user interface. I'm personally much more interested (and biased) towards automated+processed handling of the logs, and in that regard would personally just use the key-value format as an end-user. In regard to the current state of application layer logging, what is currently given in warn messages, I feel like "the agnostic data field" is the most promising for column-based logging, and that's actually what it already kind of looks like, given how the last columns of TCP are the TCP flag values, the last columns of ICMP are the ICMP Type and Code values and so on. |
|
My "universal logger" suggestion was a proposition of implementation for healthily supporting the "different possible formats" feature. It's just meant so
But I might have misunderstood the code, it does seem aimed towards having separate implementations of the Logger trait which seems like a nice structure. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi everyone,
That project seems interesting. However, can logs be enhanced to get a proper format to ingest it into whatever {ELK, Splunk, whatever} platform?
At least, mandatory things would be:
Just a quick search and I found this which might be interesting : https://rust-lang-nursery.github.io/rust-cookbook/development_tools/debugging/config_log.html
Thanks for that project, looking forward to it.
The text was updated successfully, but these errors were encountered: