WIP: Profile data mirroring

[GeigerJ2]

Design questions:

• If a Node itself is not created newly, but only added to a new group, and the mirroring is run, it is not dumped, as the node is not new... does adding to a group affect its mtime? Possibly not. Other way to get this change, e.g., check collections since the last dump?
• If a sub-workflow/calculation of another workflow is put into its own group, no folder for this group is created with the default settings
• When I delete a group, but don't delete its nodes, and I re-run the mirroring, this change is not picked up, as the filtering is done by the mtime of the node -> Currently working on this.
• Three dumping modes:
  • incremental=False, overwrite=False: Will error out with FileExistsError if the directory exists, goes through if doesn't exist or empty.
  • incremental=True, overwrite=False: Will keep original main directory, but update subdirectories with new data.
  • incremental=False, overwrite=True: Will clean main directory and perform the dumping again from scratch.
  • If both options are set to True, --overwrite will take precedence, and a report message will be issued to the user. This is because --incremental is by default True, as it is the most sensible option, and should not be required to be always specified. However, if also --overwrite is set, we don't raise an exception (as I had it initially implemented), as that would require the user to always pass --overwrite --no-incremental, which is annoying. Automatically setting --incremental to False if --overwrite is specified could be handled by a click callback, but for now I just change the options on the fly at a later stage in the code.
• Ways to specify the output path for dumping/mirroring (of all, processes, groups, and the profile data):
  • Passing no value should generate a sensible default output path in all cases.
  • If a relative path is given, this should be created under the CWD.
  • If an absolute path is given, this should be used as the top-level directory of the dumping.
  • These three options should be handled in the same way via the verdi CLI, as well as the top-level dump method of each Dumper class, so that the path can be set accordingly via the Python API.
  • To achieve this, internally, the path is split into the dump_parent_path (absolute, defaults to CWD) and an output_path part (relative, either provided by the user, or automatically generated), which, combined, yield the full top-level path where the files are dumped.

General notes:

• What happens if I delete a calculation that was called by another workchain, from AiiDA's DB, and I run with the --delete-missing option? -> Possibly use graph_traversal_rules like for verdi node delete when updating directories after a node was deleted.
• What to do if group gets deleted and verdi profile mirror --delete-missing is executed? Should also keep track of the groups in the DumpLogger, and delete the directory in that case.
• dump_parent_path is the CWD from which the dumping/mirroring command is called, while dump still provides an output_path parameter to denote the directory name of the profile, group, or process that will be dumped. This is optional, and if not provided by the user, it will be auto-generated.
• Possibly use graph_traversal_rules and add get_nodes_dump to src/aiida/tools/graph/graph_traversers.py, as well as AiidaEntitySet from src/aiida/tools/graph/age_entities.py, etc., to first obtain the nodes, and then run the dumping.

(Possible) future TODOs:

• Allow specifying options via config file
• Add data dumping (raw/rich)
• Support batch operations?
• Keep track of symlinks and/or dumped node types in DumpLogger?
• Expose endpoint to CollectionDumper and allow for mixed node types
• Add option to check directory existence/mtime/contents to determine what to do for incremental dumping

Bugs

• README for dumped processes in wrong (too high) directory