Remotely enable fleetd debug logging

[gillespi314]

Problem

User story
As a IT admin working to debug issues related to Fleet's agent,
I want to dynamically turn on debug logging for a particular host or team
so that I can obtain critical information and collaborate with Fleet engineers in resolving issues.

What have you tried?

Debug logging for Fleet's agent is controlled by command line flags for the fleetd process. Existing processes require building a new fleetd package with the flag enabled or taking ad hoc actions directly on the host (e.g., modifying launch daemons, restarting the fleetd process with the flag enabled, etc.). The process is further complicated in cases where fleetd is deployed using the fleetd-base package, such as installing fleetd via MDM, and especially so when trying to debug issues involving the macOS setup experience.

Potential solutions

Modify fleetd to allow IT admins to dynamically enable debug logging on already deployed hosts. To accomplish this, a new setting would need to be exposed via UI/API. Some ideas:

• Per host debug logging:
  • Add "Enable debug logging" to the host actions menu, which would be backed by an API endpoint that could be also by called by IT admins outside of the UI
  • Modify fleetd to look for enable_debug as part of the notifications payload
• Per team debug logging
  • Add "Enable debug logging" to the team settings, which would be backed by an API endpoint that could be also by called by IT admins outside of the UI; alternatively, this setting could be embedded into agent_options (although IMO it would be more user-friendly as a standalone setting).
  • Modify fleetd to look for enable_debug as part of the notifications (alternatively, agent_options) payload

What is the expected workflow as a result of your proposal?

After enabling the setting as described above, the IT admin would then be able to easily access the enhanced debug logs by querying a host's fleetd_logs table or inspecting the logs directly on the host.

[gillespi314]

Related to #25671

[noahtalerman]

Just chatted with @georgekarrv, and we think we need to turn on debug loggin in order to solve eponym's bug.

The plan is to add a new debug_logging agent option which turns on debug logging w/o an agent restart.

Scheduled a special design review tomorrow to update this user story and get it moving.

[noahtalerman]

@lukeheath heads up, we have this one P1 because it's blocking a workflow blocking bug: #25671

See the plan/next steps here.

[lukeheath]

@noahtalerman Makes sense, thanks!

[noahtalerman]

FYI @lukeheath @georgekarrv I removed P1 for this story but left it on the drafting board (prioritized).

It sounds like we're growing confident we have a fix for customer-eponym's bug. More info here in Slack.