Refactoring the auto generated relative paths

[stuartmcalpine]

• Add .gen_paths directory as a prefix for all auto generated relative_paths. Users therefore cannot specify a manual relative path starting with this directory, however they are they free to specify any other relative path without having to worry about clashes with auto generated names

[JoanneBogart]

I took a look at your implementation for excluding paths starting with .gen_paths. I had something much simpler in mind: just exclude any string with starts with .gen_paths, no need to break it up into path components. Certainly we need to exclude a regular file called .gen_paths which I don't think is handled by your code, but to avoid confusion, even though it's not strictly necessary, I'd also like to exclude .gen_paths.hdf or even .gen_paths_user/some_file.hdf. I don't think it will cause problems for users; they have plenty of other naming choices.

[stuartmcalpine]

I took a look at your implementation for excluding paths starting with .gen_paths. I had something much simpler in mind: just exclude any string with starts with .gen_paths, no need to break it up into path components. Certainly we need to exclude a regular file called .gen_paths

Changes should now reflect this. I've added some more checks also for the relative_path in general (in dataset.py). Including illegal chars, but lmk if this is still the behavior you want here.

Users should not be able to put anything into the .gen_paths dir, or chose the relative_path just as .gen_paths

I'd also like to exclude .gen_paths.hdf or even .gen_paths_user/some_file.hdf. I don't think it will cause problems for users; they have plenty of other naming choices.

Not added this yet, any reason about .hdf?

[stuartmcalpine]

I've changed the way single files are ingested to preserve filename suffixes.

Now when a single file is ingested, the autogenerated part (i.e., the directory with name and version) is reserved for the directory name (within .gen_paths) and the file is simply copied into the folder, preserving the filename. e,g., something like .gen_paths/mydataset_1.0.0/this_was_the_filename.txt

[JoanneBogart]

LGTM!
Thanks