[Feature Request] Logging required for RedELK integration

[MarcOverIP]

Feature Request
We would want to have Covenant log its actions to a log file so we can work on RedELK integration (https://github.com/outflanknl/RedELK)
Covenant currently has some data in its sqlite db. But the info is not really sufficient and its not really doable for RedELK to parse that file in search for relevant logs.

Expected behavior
Ideally the logging by Covenant is done 1) text based, 2) follows a structured setup per log, 3) does timestamps in UTC, 4) logs operator actions, as well as relevant other actions such as keystrokes and screenshots received.

Overall I could say that if you follow the same approach as how Cobalt Strike does its logging, we should be good.

Additional context

• More info on RedELK: https://github.com/outflanknl/RedELK
• The approach that we have on the RedELK side: Support for Covenant C2 framework outflanknl/RedELK#23
• An example of how PoshC2 went from basic logging to something we can use in RedELK: Support for PoshC2 outflanknl/RedELK#22
• More info on the goals of RedELK and the way it works in this multiple part blog series: https://outflank.nl/blog/2019/02/14/introducing-redelk-part-1-why-we-need-it/ , https://outflank.nl/blog/2020/02/28/redelk-part-2-getting-you-up-and-running/ and https://outflank.nl/blog/2020/04/07/redelk-part-3-achieving-operational-oversight/

[checkymander]

So Covenant has an "Event Hub" that can be subscribed to, which is a websocket that pushes events happening out to everyone who's subscribed. Is that something that could be used, or does it have to be written out to a log file?

[MarcOverIP]

As far as RedELK goes I would like keep the C2 adapters as simple and default as possible. That means that for current supported C2 frameworks PoshC2 we ingest text based logs with a simple filebeat config (default package part of the Elastic stack). Now, Filebeat can work with txt, json or whatever txt based logging. Websocket is not supported.

Theoretically we could make a Covenant specific adapter with whatever fancy code we would like, eg websocket based. But that would 1) deviate from the setup of other C2s, and 2) not placed best at the hands of RedELK developers as we would always be behind the pace of development of Covenant. Ideally this (the fact there is detailed logging to log files) is regarded as new feature of Covenant itself. We as RedELK developers will have a busy time already making sure all the data is properly ingested and indexed within RedELK itself.

[fastlorenzo]

If such "Event Hub" already exists, shouldn't it be possible to integrate a listener in Covenant that writes event to a log file?

[checkymander]

Digging into the code it looks like the concept of logging does exist, however the only time it outputs is if there's a fatal error when starting a listener/covenant. @cobbr, do you have thoughts on expanding the existing loggers to be covenant-wide and include more logging (e.g. Initial Grunt/Brute Callback, Grunt/Brute tasking, etc etc)?

[cobbr]

I agree that it makes more sense for Covenant to implement better logging than for redelk to try to turn websockets events into a coherent log.

It's going to take some time to fine tune the logging just right. The last time I tried to enable logging, the logs would eat up all available hard drive space in a matter of days, lol.

[MarcOverIP]

Please don't overdo it ;-)

If you can get the same level of details as Cobalt Strike does, it should be good!

[checkymander]

Threw together an initial PR to support logging for this Issue. PR #244, take a look and let me know what you'd like added. Not quite the same level of detail as a CS beacon, but should be a start.

[MarcOverIP]

Awesome! Do you have any examples of the logs it generates?

Also paging @fastlorenzo , you may want to take a loot at this

[fastlorenzo]

Really nice, indeed! Let's check what you have in this PR for now and then we'll see if some other things needs to be added/adapted to get an exhaustive view in RedELK 👍

[checkymander]

Here's a general idea of what the logs output on specific covenant events. I'm open to changing them if that would make parsing easier for you guys or if @cobbr has a preference. By default the PR doesn't include Command Output but the code exists and can be un-commented out if someone wanted to include that in their logs.

2020-09-03 16:43:20.8089||INFO|Covenant.Core.LoggingService|[OnCreateCovenantUser] ab10a915-3cee-4bf9-9cca-699bfed31fe1 - 6E3C6A15-B39F-BF13-82FD-BD2456FD1DE2  
2020-09-03 16:43:20.9877||INFO|Covenant.Core.LoggingService|[OnCreateListener] ListenerID: 1
	BindAddress: 0.0.0.0
	BindPort: 80
	ConnectAddresses: 10.10.0.4
	ConnectPort: 80 
2020-09-19 15:23:24.7490||INFO|Covenant.Core.LoggingService|[OnCreateCovenantUser] 0e89d748-37a5-4201-8a12-f1d4e28ce457 - 44F2CAEC-92A6-C1F2-F4B2-D302936C815F 
2020-09-19 15:23:24.9657||INFO|Covenant.Core.LoggingService|[OnCreateListener] ListenerID: 1
	BindAddress: 0.0.0.0
	BindPort: 80
	ConnectAddresses: 10.10.0.2
	ConnectPort: 80 
2020-09-19 15:23:38.4850||INFO|Covenant.Core.LoggingService|[OnCreateHostedFile] ID: 1
	Path: /path/to/file.exe 
2020-09-19 15:26:28.8140||INFO|Covenant.Core.LoggingService|[OnCreateGrunt] 5 
	e840756b32
	DESKTOP-CQRINT5
	Medium
	10.10.0.2
	DESKTOP-CQRINT5 

I should probably update the OnCreateGrunt log message to include the property names to be consistent too, so that will probably be a change that's coming.

[fastlorenzo]

I think it would be easier to parse in logstash with something else than a newline delimited (if possible for you) 👍

Regarding the logged data, what would be super interesting to have is everything that happens in the target network: tasks launched by operator + output, downloads, retrieved credentials, screenshots, keylogger data, etc.

[fastlorenzo]

Something like this could be more convenient for parsing:

2020-09-03 16:43:20.8089||INFO|Covenant.Core.LoggingService|[OnCreateCovenantUser]|ab10a915-3cee-4bf9-9cca-699bfed31fe1|6E3C6A15-B39F-BF13-82FD-BD2456FD1DE2  
2020-09-03 16:43:20.9877||INFO|Covenant.Core.LoggingService|[OnCreateListener]|ListenerID:1|BindAddress:0.0.0.0|BindPort:80|ConnectAddresses:10.10.0.4|ConnectPort:80 
2020-09-19 15:23:24.7490||INFO|Covenant.Core.LoggingService|[OnCreateCovenantUser]|0e89d748-37a5-4201-8a12-f1d4e28ce457|44F2CAEC-92A6-C1F2-F4B2-D302936C815F
2020-09-19 15:23:24.9657||INFO|Covenant.Core.LoggingService|[OnCreateListener]ListenerID:1|BindAddress:0.0.0.0|BindPort:80|ConnectAddresses:10.10.0.2|ConnectPort:80
2020-09-19 15:23:38.4850||INFO|Covenant.Core.LoggingService|[OnCreateHostedFile]|ID:1|Path:/path/to/file.exe 
2020-09-19 15:26:28.8140||INFO|Covenant.Core.LoggingService|[OnCreateGrunt]|5|e840756b32|DESKTOP-CQRINT5|Medium|10.10.0.2|DESKTOP-CQRINT5 

[checkymander]

I agree, I think that output does look better, I'll work on changing that. Currently the logging does support new credentials, and tasks created by operators, however after discussing with @cobbr we decided on not including command output by default. However, I do have the code in there, which can be un-commented out if someone wanted to implement it. I've basically included all of the "important" data outside of the snippet I pasted here.

[fastlorenzo]

That's great, thanks! I'll give it a try tomorrow and test parsing the output.

Why did you decide not to log the command output?

[checkymander]

Just a lot of potential customer data being stored in plaintext logs that I don't really want to make it "the default". The code is still there and can be enabled, but I'd rather leave that up to the users.

[fastlorenzo]

makes sense, could we make this a configuration setting instead? Just to avoid having the users needing to change the code itself to make it work 😄

[checkymander]

I'll take a look when I have a bit of time, I'm currently revamping the logging logic so I'll look into that as well.

[cobbr]

Alright, so we finally have some movement on logging. Merged in @checkymander's PR, with a slightly altered log format if you want to start to see how it might fit in with RedELK @MarcOverIP

[MarcOverIP]

Great, thank you. Whats the quickest way to get some example logs for us to work with?

/cc @fastlorenzo

[cobbr]

@MarcOverIP Here's a few random samples. Might also be helpful to check the references for ILoggable:

Covenant/Covenant/Models/Common.cs

Line 12 in 5384760

public interface ILoggable

covenant_2021-02-05.00.log
covenant_2021-02-08.00.log
covenant_2021-02-09.00.log
covenant.log

[MarcOverIP]

Thank you, this helps to get started and very much looks like something that we can work with!

We'll take this up when we closed off some pending issues on our side. Feel free to ping us if this issue remains open longer than you feel comfortabel with :-)

[cobbr]

Yep, no rush. I sure took my time on it, so don't feel like you have to rush for my sake.

[fastlorenzo]

Awesome, thanks @cobbr !