Wait for a command to stop before shutting down

[schomatis]

This problem was manifested in #4820 which I'll briefly describe here for context but actually exceeds its use case: while using the Badger datastore in offline mode, if the user cancels an add command there is a (very likely) possibility that the Badger DB is closed while open transactions are still operating on it (causing a panic as the Badger API does not allow such scenario but also doesn't take measures to prevent it, leaving that responsibility to the consumer of the API). The user is pressing Ctrl-C triggering the IpfsNode.teardown() procedure which proceeds to close the BlockService (and its inherent Datastore which is backed in this case by a Badger DB). In the meanwhile the add command keeps issuing store requests (which conflict with the close request from the BlockService shutdown causing the panic). Ideally the shutdown procedure should wait on the add command to finish (which also received a cancellation signal from the Ctrl-C but might take more time to act on it than teardown()) before closing the services on which the command relies.

As @Kubuxu mentioned this problem touches on other already existing issues (and I think it can have potential repercussions on others as well) so while my primary focus is to provide a solution for #4820 I'm opened to a more a comprehensive solution to improve inter-service communications (I'm considering the command dispatching a service although I'm not sure if it has that status).

I'm keeping this issue as the central reference for other issues/PRs addressing it (some of the will be in the go-ipfs-cmds repo).

[Kubuxu]

@schomatis in short: there is ipfs shutdown command that shuts down the node. The problem is that the node shutdown sometimes happens before or after the response from this command reaches CLI client thus giving an error on the CLI.

[schomatis]

@Kubuxu Exactly, the ipfs shutdown command in online mode is the analog of Ctrl-Cing during command execution in offline mode, and the lack of communication between the core node shutdown and the service dispatching the HTTP response for the CLI client triggers a similar error (cause) to the Badger scenario.

[schomatis]

As a first step towards a solution, as mentioned in #4820, the ReqLog can be extended to cover the offline case (see ipfs/go-ipfs-cmds#109) and have it as a bookkeeping control structure to know when a command has finished. (I'm not sure if we are abusing this structure way beyond it's original informative purpose without the responsibility of managing control flow, was that the original intention for the request log?)