Application profiling

[jw3]

Provide ability to run arbitrary applications with fapolicyd in permissive debug mode while capturing the log for analysis.

1. Select "Run profiler" from menu
2. A dialog pops up to enter a command (free text, maybe a file picker)
3. Select "Start"
4. Stop fapolicyd daemon service (if it is running)
5. Start fapolicyd daemon in subprocess with permissive + debug modes, dumping output to temp file
6. When application is complete start the daemon service again (if it was running)
7. Parse and display the temp file from debug output in the analyzer view

Determining when the application is "complete" can be tricky in some cases, like with a GUI application that will open a window and may require interaction. Interaction seems fine, just need to make sure the subprocess termination is detected in all cases so it doesn't hang.

The goal here is to make it easy to identify what entries are required for an application to function.

[tparchambault]

Determining when the application is "complete" can be tricky in some cases, like with a GUI application that will open a window and may require interaction. Interaction seems fine, just need to make sure the subprocess termination

Agreed. This may not be possible in the general case for all applications that are comprised of many execution threads/processes/plug-ins paths e.g. different codecs loaded depending on the input data stream. Maybe setting expectations such that multiple iterations may be required to cover different use cases. Don't have a real solution other than to make it easy for the administrator to quickly alternate between user activities and analysis activities when performed manually. Internal execution automation or scripting might be something to consider down the road, allowing sequential runs of an application with different input data while capturing fapolicyd events for those runs.

[jw3]

I think it makes sense to have an initial run at this functionality without the GUI, as a CLI application that generates the log, we can load it in the app as a separate step for now. The implementation of profiling components should be with importing them into a CLI or GUI structure in mind, either type of application should be able to compose and execute the profiler.

The implementation should go under a new package at fapolicy_analyzer.profiler or some such. @dorschs57 thoughts?

[tparchambault]

Off the top of my head, I thought that we can have a new systemd service file with any new runtime settings that differ from those of an on-line session of fapolicyd. Then the existing backend comm w/systemd can be be reused except wrt the new 'service', fapolicyd-profiliing.service, or whatever we want to call it.

Logging options for the profiling session could be distinct from those of the on-line session, and if desired, we could run the profiling session in a higher security sandbox by using systemd's kernel capabilities, seccomp, namespace features (or none at all.) And of course, the Exec line will have the permissive option.

When the application transitions into profiling mode, we stop fapolicyd.service via systemd, then start fapolicyd-profiling.service. and perform the inverse on the transition back to on-line mode (or whatever we want to call that mode.)

[jw3]

What advantage is there to use systemd instead of using a sub process?

What downside is there to parsing system logs vs stdout logs?

Do you have to clear the unit logs between runs?

Who owns this systemd unit we are installing on the machine?

[tparchambault]

What advantage is there to use systemd instead of using a sub process?

We would be using the same control plumbing that we already use to control fapolicyd albeit w/different options and w/o modifying the on-line infrastructure. We pick up a lot of potential functionality w/o writing code. We use systemd now to start and stop the daemon; for a profiling session, we'd be then starting and stopping the daemon again with different options and possibly in a different environment if that is beneficial. And that's all free by just specifying it in a service file.

It's an incremental change to implement this functionality through a service file. We are not modifying the existing shipped service file, and we don't have to leverage any security or logging options to start but can if/when they make sense. I'm also wrestling with the question, "should test/profiling runs be allowed to modify files?" So was thinking about read-only fs w/overlays. But again, out of the box, we don't need to do anything except add the permissive option.

What downside is there to parsing system logs vs stdout logs?

I do not know. You are much more aware of the logging/parsing processing than I. We can use the same default logging mechanisms but also have options if we chose to exercise them. I believe, we can tag the profiling service logging differently than the standard fapolicyd tagging, so may be easier to extract from the logs if we maintain the current logging mechanisms.

My picture is that whatever logging we generate would be stand-alone and time-limited, i.e. we would not be extracting from the on-line logging data, nor would the on-line logging data be polluted w/extraneous and irrelevant events that occur during testing. But again, there's no reason we can't continue to send them to the same logs.

Do you have to clear the unit logs between runs?

I'm thinking they reside in a tmp directory and we would throw them away after the analysis work is completed or give the user the option of saving them. I think the systemd unit file can provide a dedicated tmp directory for a service. Just to be clear we are not running two instances of fapolicyd at the same time.

Who owns this systemd unit we are installing on the machine?

fapolicy-analyzer, whether the service file is shipped and installed, or we dynamically create it and install it under /etc/systemd/ (iirc). The fapolicyd dev team has no idea what constitutes profiling from the fapolicy-analyzer perspective nor should they IMO.

[tparchambault]

What advantage is there to use systemd instead of using a sub process?

I might have missed your point about the subprocess. Everything I wrote above was fapolicyd specific: systemd is only controlling the profiling instance of fapolicyd. The application under test/profiling (AUP) will be managed as a subprocess, and your concerns stated above about determining when the application is complete remain valid. However, it may be useful to consider running the AUP within a systemd environment/sandbox. I need to look at that more.

[jw3]

I am somewhat concerned about the additional overhead of the systemd unit vs a subprocess. Though I am interested in seeing what advanced features systemd can provide as far as sandboxing that might be beneficial to us.

I like the aspects of

1. generating and installing the unit file on-demand so that we can control properties in it to make unique runs
2. isolating logs from the rest of the system
3. leveraging temp directories or chroot-like capabilities

Let's see how the systemd way looks.