drivers: update ordering of events in StartTask to fix executor leak

[mismithhisler]

Description

If an error occurs before a task is started, but after this executor process is created, the executor must be explicity stopped. This change updates the logic in StartTask so launching the task happens immediately after creating the executor.

Testing & Reproduction steps

Run a job with a configuration that would fail after the executor process has started, but before the task is launched. For example, an exec task with cap_add = ["net_raw"]. After the job fails, the executor process will still be running.

Links

Fixes GH #11958

Contributor Checklist

• Changelog Entry If this PR changes user-facing behavior, please generate and add a
  changelog entry using the make cl command.
• Testing Please add tests to cover any new functionality or to demonstrate bug fixes and
  ensure regressions will be caught.
• Documentation If the change impacts user-facing functionality such as the CLI, API, UI,
  and job configuration, please update the Nomad website documentation to reflect this. Refer to
  the website README for docs guidelines. Please also consider whether the
  change requires notes within the upgrade guide.

Reviewer Checklist

• Backport Labels Please add the correct backport labels as described by the internal
  backporting document.
• Commit Type Ensure the correct merge method is selected which should be "squash and merge"
  in the majority of situations. The main exceptions are long-lived feature branches or merges where
  history should be preserved.
• Enterprise PRs If this is an enterprise only PR, please add any required changelog entry
  within the public repository.

[mismithhisler]

I wonder if using named return values would be a better way to prevent a regression. With named return values, we can defer a function that checks if err != nil, and kill the plugin client. This way we don't have to worry about accidentally leaving the executor process running in any added codepaths in the future.

We could also write a single test that forces any error after the executor is created, and that should suffice to catch this bug in most other error scenarios.

[jrasell]

I wonder if using named return values would be a better way to prevent a regression. With named return values, we can defer a function that checks if err != nil, and kill the plugin client. This way we don't have to worry about accidentally leaving the executor process running in any added codepaths in the future.

We could also write a single test that forces any error after the executor is created, and that should suffice to catch this bug in most other error scenarios.

If you think it's a solution worth exploring, then I would encourage you to look into it. The testing aspect in particular would be very useful in my opinion. The current code in this PR looks like it would fix the problem, so we can always fallback here if needed.

[jrasell]

LGTM and thanks @mismithhisler!

I tested this on a Linux workstation with the details supplied and it was easy to witness the fix.

I noticed the added test takes 12s to run which seems a little long. It might be a follow up to see if this can be lowered. Interestingly, the NoOrphanedTasks test is also long at 27s, so it might be something these tests need.