diff --git a/previews/PR71/design/index.html b/previews/PR71/design/index.html index 0bee0f8..5d4e4b3 100644 --- a/previews/PR71/design/index.html +++ b/previews/PR71/design/index.html @@ -4,4 +4,4 @@ open(joinpath(git_tree, "some_blob.txt"), write=true) do io write(io, "hi") end -end

There's at least two quite different use patterns for versioning:

Provenance: What is this data? What was I thinking?

Working with historical data can be confusing and error prone because the origin of that data may look like this:

[xkcd 1838](https://xkcd.com/1838)

The solution is to systematically record how data came to be, including input parameters and code version. This data provenance information comes from your activity as encoded in a possibly-interactive program, but must be stored alongside the data.

A full metadata system for data provenance is out of scope for DataSets.jl — it's a big project in its own right. But I think we should arrange the data lifecycle so that provenance can be hooked in easily by providing:

Some interesting links about provenance metadata:

Data Models

The Data Model is the abstraction which the dataset user interacts with. In general this can be provided by some arbitrary Julia code from an arbitrary module. We'll need a way to map the DataSet into the code which exposes the data model.

Examples, including some example storage formats which the data model might overlay

Distributed and incremental processing

For distributed or incremental processing of large data, it must be possible to load data lazily and in parallel: no single node in the computation should need the whole dataset to be locally accessible.

Not every data model can support efficient parallel processing. But for those that do it seems that the following concepts are important:

To be clear, DataSets largely doesn't provide these things itself — these are up to implementations of particular data models. But the data lifecycle should be designed to efficiently support distributed computation.

Tree-indexed data

This is one particular data model which I've tackle this as a first use case, as a "hieracical tree of data" is so common. Examples are

But we don't have a well-defined path tree abstraction which already exists! So I've been prototyping some things in this package. (See also FileTrees.jl which is a new and very recent package tackling similar things.)

Paths and Roots

What is a tree root object? It's a location for a data resource, including enough information to open that resource. It's the thing which handles the data lifecycle events on the whole tree.

What is a relative path, in general? It's a key into a heirarchical tree-structured data store. This consists of several path components (an array of strings)

Iteration

Interesting related projects

+end

There's at least two quite different use patterns for versioning:

Provenance: What is this data? What was I thinking?

Working with historical data can be confusing and error prone because the origin of that data may look like this:

[xkcd 1838](https://xkcd.com/1838)

The solution is to systematically record how data came to be, including input parameters and code version. This data provenance information comes from your activity as encoded in a possibly-interactive program, but must be stored alongside the data.

A full metadata system for data provenance is out of scope for DataSets.jl — it's a big project in its own right. But I think we should arrange the data lifecycle so that provenance can be hooked in easily by providing:

Some interesting links about provenance metadata:

Data Models

The Data Model is the abstraction which the dataset user interacts with. In general this can be provided by some arbitrary Julia code from an arbitrary module. We'll need a way to map the DataSet into the code which exposes the data model.

Examples, including some example storage formats which the data model might overlay

Distributed and incremental processing

For distributed or incremental processing of large data, it must be possible to load data lazily and in parallel: no single node in the computation should need the whole dataset to be locally accessible.

Not every data model can support efficient parallel processing. But for those that do it seems that the following concepts are important:

To be clear, DataSets largely doesn't provide these things itself — these are up to implementations of particular data models. But the data lifecycle should be designed to efficiently support distributed computation.

Tree-indexed data

This is one particular data model which I've tackle this as a first use case, as a "hieracical tree of data" is so common. Examples are

But we don't have a well-defined path tree abstraction which already exists! So I've been prototyping some things in this package. (See also FileTrees.jl which is a new and very recent package tackling similar things.)

Paths and Roots

What is a tree root object? It's a location for a data resource, including enough information to open that resource. It's the thing which handles the data lifecycle events on the whole tree.

What is a relative path, in general? It's a key into a heirarchical tree-structured data store. This consists of several path components (an array of strings)

Iteration

Interesting related projects

diff --git a/previews/PR71/index.html b/previews/PR71/index.html index 4bca413..8af211c 100644 --- a/previews/PR71/index.html +++ b/previews/PR71/index.html @@ -1,2 +1,2 @@ -Introduction · DataSets.jl
Edit on GitHub

DataSets.jl

DataSets.jl exists to help manage data and reduce the amount of data wrangling code you need to write. It's annoying to constantly rewrite

  • Command line wrappers which deal with paths to data storage
  • Code to load and save from various data storage systems (eg, local filesystem data; local git data, downloaders for remote data over various protocols, cloud storage access, proprietary data servers, etc)
  • Code to load the same data model from various serializations
  • Code to deal with data lifecycle; versions, provenance, etc

DataSets provides scaffolding to make this kind of code more reusable. We want to make it easy to relocate an algorithm between different data environments without code changes. For example from your laptop to the cloud, to another user's machine, or to an HPC system.

DataSets.jl is in early development! We're still figuring out the basic shape of the design. So things will change, but your input is important: we need your use cases so that the design serves the real needs of people dealing with data.

+Introduction · DataSets.jl
Edit on GitHub

DataSets.jl

DataSets.jl exists to help manage data and reduce the amount of data wrangling code you need to write. It's annoying to constantly rewrite

  • Command line wrappers which deal with paths to data storage
  • Code to load and save from various data storage systems (eg, local filesystem data; local git data, downloaders for remote data over various protocols, cloud storage access, proprietary data servers, etc)
  • Code to load the same data model from various serializations
  • Code to deal with data lifecycle; versions, provenance, etc

DataSets provides scaffolding to make this kind of code more reusable. We want to make it easy to relocate an algorithm between different data environments without code changes. For example from your laptop to the cloud, to another user's machine, or to an HPC system.

DataSets.jl is in early development! We're still figuring out the basic shape of the design. So things will change, but your input is important: we need your use cases so that the design serves the real needs of people dealing with data.

diff --git a/previews/PR71/reference/index.html b/previews/PR71/reference/index.html index 9dd7626..2ae0700 100644 --- a/previews/PR71/reference/index.html +++ b/previews/PR71/reference/index.html @@ -3,13 +3,13 @@ dataset(project, name)

Returns the DataSet with the given name from project. If omitted, the global data environment DataSets.PROJECT will be used.

The DataSet is metadata, but to use the actual data in your program you need to use the open function to access the DataSet's content as a given Julia type.

Example

To open a dataset named "a_text_file" and read the whole content as a String,

content = open(String, dataset("a_text_file"))

To open the same dataset as an IO stream and read only the first line,

open(IO, dataset("a_text_file")) do io
     line = readline(io)
     @info "The first line is" line
-end

To open a directory as a browsable tree object,

open(BlobTree, dataset("a_tree_example"))
source
DataSets.@datafunc — Macro
@datafunc function f(x::DT=>T, y::DS=>S...)
+end

To open a directory as a browsable tree object,

open(BlobTree, dataset("a_tree_example"))
source
DataSets.@datafunc — Macro
@datafunc function f(x::DT=>T, y::DS=>S...)
     ...
-end

Define the function f(x::T, y::S, ...) and add data dispatch rules so that f(x::DataSet, y::DataSet) will open datasets matching dataset types DT,DS as Julia types T,S.

source
DataSets.@datarun — Macro
@datarun [proj] func(args...)

Run func with the named DataSets from the list args.

Example

Load DataSets named a,b as defined in Data.toml, and pass them to f().

proj = DataSets.load_project("Data.toml")
-@datarun proj f("a", "b")
source

Data environment

The global data environment for the session is defined by DataSets.PROJECT which is initialized from the JULIA_DATASETS_PATH environment variable. To load a data project from a particular TOML file, use DataSets.load_project.

DataSets.PROJECT — Constant

DataSets.PROJECT contains the default global data environment for the Julia process. This is created from the JULIA_DATASETS_PATH environment variable at initialization which is a list of paths (separated by : or ; on windows).

In analogy to Base.LOAD_PATH and Base.DEPOT_PATH, the path components are interpreted as follows:

  • @ means the path of the current active project as returned by Base.active_project(false) This can be useful when you're "doing scripting" and you've got a project-specific Data.toml which resides next to the Project.toml. This only applies to projects which are explicitly set with julia --project or Pkg.activate().
  • Explicit paths may be either directories or files in Data.toml format. For directories, the filename "Data.toml" is implicitly appended. expanduser() is used to expand the user's home directory.
  • As in DEPOT_PATH, an empty path component means the user's default Julia home directory, joinpath(homedir(), ".julia", "datasets")

This simplified version of the code loading rules (LOADPATH/DEPOTPATH) is used as it seems unlikely that we'll want data location to be version- dependent in the same way that that code is.

Unlike LOAD_PATH, JULIA_DATASETS_PATH is represented inside the program as a StackedDataProject, and users can add custom projects by defining their own AbstractDataProject subtypes.

Additional projects may be added or removed from the stack with pushfirst!, push! and empty!.

source
DataSets.load_project — Function
load_project(path; auto_update=false)
-load_project(config_dict)

Load a data project from a system path referring to a TOML file. If auto_update is true, the returned project will monitor the file for updates and reload when necessary.

Alternatively, create a DataProject from a an existing dictionary config_dict, which should be in the Data.toml format.

See also load_project!.

source
DataSets.load_project! — Function
load_project!(path_or_config)

Prepends to the default global dataset search stack, DataSets.PROJECT.

May be renamed in a future version.

source

DataSet metadata model

The DataSet is a holder for dataset metadata, including the type of the data and the method for access (the storage driver - see Storage Drivers). DataSets are managed in projects which may be stacked together. The library provides several subtypes of DataSets.AbstractDataProject for this purpose which are listed below. (Most users will simply to configure the global data project via DataSets.PROJECT.)

DataSets.DataSet — Type

A DataSet is a metadata overlay for data held locally or remotely which is unopinionated about the underlying storage mechanism.

The data in a DataSet has a type which implies an index; the index can be used to partition the data for processing.

source
DataSets.AbstractDataProject — Type

Subtypes of AbstractDataProject have the interface

Must implement:

  • Base.get(project, dataset_name, default) — search
  • Base.keys(project) - get dataset names

Optional:

  • Base.iterate() — default implementation in terms of keys and get
  • Base.pairs() — default implementation in terms of keys and get
  • Base.haskey() — default implementation in terms of get
  • Base.getindex() — default implementation in terms of get
  • DataSets.project_name() — returns nothing by default.

Provided by AbstractDataProject (should not be overridden):

  • DataSets.dataset() - implemented in terms of get
source
DataSets.DataProject — Type
DataProject

A concrete data project is a collection of DataSets with associated names. Names are unique within the project.

source
DataSets.StackedDataProject — Type
StackedDataProject()
-StackedDataProject(projects)

Search stack of AbstractDataProjects, where projects are searched from the first to last element of projects.

Additional projects may be added or removed from the stack with pushfirst!, push! and empty!.

See also DataSets.PROJECT.

source
DataSets.ActiveDataProject — Type

Data project, based on the location of the current explicitly selected Julia Project.toml, as reported by Base.active_project(false).

Several factors make the implementation a bit complicated:

  • The active project may change at any time without warning
  • The active project may be nothing when no explicit project is selected
  • There might be no Data.toml for the active project
  • The user can change Data.toml interactively and we'd like that to be reflected within the program.
source
DataSets.TomlFileDataProject — Type

Data project which automatically updates based on a TOML file on the local filesystem.

source

Data Models for files and directories

DataSets provides some builtin data models Blob and BlobTree for accessin file- and directory-like data respectively. For modifying these, the functions newfile and newdir can be used, together with setindex! for BlobTree.

DataSets.Blob — Type
Blob(root)
-Blob(root, relpath)

Blob represents the location of a collection of unstructured binary data. The location is a path relpath relative to some root data resource.

A Blob can naturally be open()ed as a Vector{UInt8}, but can also be mapped into the program as an IO byte stream, or interpreted as a String.

Blobs can be arranged into hierarchies "directories" via the BlobTree type.

source
DataSets.BlobTree — Type
BlobTree(root)

BlobTree is a "directory tree" like hierarchy which may have Blobs and BlobTrees as children.

The tree implements the AbstracTrees.children() interface and may be indexed with paths to traverse the hierarchy down to the leaves ("files") which are of type Blob. Individual leaves may be open()ed as various Julia types.

Example

Normally you'd construct these via the dataset function which takes care of constructing the correct root object. However, here's a direct demonstration:

julia> tree = BlobTree(DataSets.FileSystemRoot(dirname(pathof(DataSets))), path"../test/data")
+end

Define the function f(x::T, y::S, ...) and add data dispatch rules so that f(x::DataSet, y::DataSet) will open datasets matching dataset types DT,DS as Julia types T,S.

source
DataSets.@datarun — Macro
@datarun [proj] func(args...)

Run func with the named DataSets from the list args.

Example

Load DataSets named a,b as defined in Data.toml, and pass them to f().

proj = DataSets.load_project("Data.toml")
+@datarun proj f("a", "b")
source

Data environment

The global data environment for the session is defined by DataSets.PROJECT which is initialized from the JULIA_DATASETS_PATH environment variable. To load a data project from a particular TOML file, use DataSets.load_project.

DataSets.PROJECT — Constant

DataSets.PROJECT contains the default global data environment for the Julia process. This is created from the JULIA_DATASETS_PATH environment variable at initialization which is a list of paths (separated by : or ; on windows).

In analogy to Base.LOAD_PATH and Base.DEPOT_PATH, the path components are interpreted as follows:

  • @ means the path of the current active project as returned by Base.active_project(false) This can be useful when you're "doing scripting" and you've got a project-specific Data.toml which resides next to the Project.toml. This only applies to projects which are explicitly set with julia --project or Pkg.activate().
  • Explicit paths may be either directories or files in Data.toml format. For directories, the filename "Data.toml" is implicitly appended. expanduser() is used to expand the user's home directory.
  • As in DEPOT_PATH, an empty path component means the user's default Julia home directory, joinpath(homedir(), ".julia", "datasets")

This simplified version of the code loading rules (LOADPATH/DEPOTPATH) is used as it seems unlikely that we'll want data location to be version- dependent in the same way that that code is.

Unlike LOAD_PATH, JULIA_DATASETS_PATH is represented inside the program as a StackedDataProject, and users can add custom projects by defining their own AbstractDataProject subtypes.

Additional projects may be added or removed from the stack with pushfirst!, push! and empty!.

source
DataSets.load_project — Function
load_project(path; auto_update=false)
+load_project(config_dict)

Load a data project from a system path referring to a TOML file. If auto_update is true, the returned project will monitor the file for updates and reload when necessary.

Alternatively, create a DataProject from a an existing dictionary config_dict, which should be in the Data.toml format.

See also load_project!.

source
DataSets.load_project! — Function
load_project!(path_or_config)

Prepends to the default global dataset search stack, DataSets.PROJECT.

May be renamed in a future version.

source

DataSet metadata model

The DataSet is a holder for dataset metadata, including the type of the data and the method for access (the storage driver - see Storage Drivers). DataSets are managed in projects which may be stacked together. The library provides several subtypes of DataSets.AbstractDataProject for this purpose which are listed below. (Most users will simply to configure the global data project via DataSets.PROJECT.)

DataSets.DataSet — Type

A DataSet is a metadata overlay for data held locally or remotely which is unopinionated about the underlying storage mechanism.

The data in a DataSet has a type which implies an index; the index can be used to partition the data for processing.

source
DataSets.AbstractDataProject — Type

Subtypes of AbstractDataProject have the interface

Must implement:

  • Base.get(project, dataset_name, default) — search
  • Base.keys(project) - get dataset names

Optional:

  • Base.iterate() — default implementation in terms of keys and get
  • Base.pairs() — default implementation in terms of keys and get
  • Base.haskey() — default implementation in terms of get
  • Base.getindex() — default implementation in terms of get
  • DataSets.project_name() — returns nothing by default.

Provided by AbstractDataProject (should not be overridden):

  • DataSets.dataset() - implemented in terms of get
source
DataSets.DataProject — Type
DataProject

A concrete data project is a collection of DataSets with associated names. Names are unique within the project.

source
DataSets.StackedDataProject — Type
StackedDataProject()
+StackedDataProject(projects)

Search stack of AbstractDataProjects, where projects are searched from the first to last element of projects.

Additional projects may be added or removed from the stack with pushfirst!, push! and empty!.

See also DataSets.PROJECT.

source
DataSets.ActiveDataProject — Type

Data project, based on the location of the current explicitly selected Julia Project.toml, as reported by Base.active_project(false).

Several factors make the implementation a bit complicated:

  • The active project may change at any time without warning
  • The active project may be nothing when no explicit project is selected
  • There might be no Data.toml for the active project
  • The user can change Data.toml interactively and we'd like that to be reflected within the program.
source
DataSets.TomlFileDataProject — Type

Data project which automatically updates based on a TOML file on the local filesystem.

source

Data Models for files and directories

DataSets provides some builtin data models Blob and BlobTree for accessin file- and directory-like data respectively. For modifying these, the functions newfile and newdir can be used, together with setindex! for BlobTree.

DataSets.Blob — Type
Blob(root)
+Blob(root, relpath)

Blob represents the location of a collection of unstructured binary data. The location is a path relpath relative to some root data resource.

A Blob can naturally be open()ed as a Vector{UInt8}, but can also be mapped into the program as an IO byte stream, or interpreted as a String.

Blobs can be arranged into hierarchies "directories" via the BlobTree type.

source
DataSets.BlobTree — Type
BlobTree(root)

BlobTree is a "directory tree" like hierarchy which may have Blobs and BlobTrees as children.

The tree implements the AbstracTrees.children() interface and may be indexed with paths to traverse the hierarchy down to the leaves ("files") which are of type Blob. Individual leaves may be open()ed as various Julia types.

Example

Normally you'd construct these via the dataset function which takes care of constructing the correct root object. However, here's a direct demonstration:

julia> tree = BlobTree(DataSets.FileSystemRoot(dirname(pathof(DataSets))), path"../test/data")
 📂 Tree ../test/data @ /home/chris/.julia/dev/DataSets/src
  📁 csvset
  📄 file.txt
@@ -24,7 +24,7 @@
 julia> tree[path"csvset"]
 📂 Tree ../test/data/csvset @ /home/chris/.julia/dev/DataSets/src
  📄 1.csv
- 📄 2.csv
source
DataSets.newfile — Function
newfile(func)
+ 📄 2.csv
source
DataSets.newfile — Function
newfile(func)
 newfile(func, ctx)

Create a new temporary Blob object which may be later assigned to a permanent location in a BlobTree. If not assigned to a permanent location, the temporary file is cleaned up during garbage collection.

Example

tree[path"some/demo/path.txt"] = newfile() do io
     println(io, "Hi there!")
-end
source
DataSets.newdir — Function
newdir()

Create a new temporary BlobTree which can have files assigned into it and may be assigned to a permanent location in a persistent BlobTree. If not assigned to a permanent location, the temporary tree is cleaned up during garbage collection.

source

Storage Drivers

To add a new kind of data storage backend, implement DataSets.add_storage_driver

DataSets.add_storage_driver — Function
add_storage_driver(driver_name=>storage_opener)

Associate DataSet storage driver named driver_name with storage_opener. When a dataset with storage.driver == driver_name is opened, storage_opener(user_func, storage_config, dataset) will be called. Any existing storage driver registered to driver_name will be overwritten.

As a matter of convention, storage_opener should generally take configuration from storage_config which is just dataset.storage. But to avoid config duplication it may also use the content of dataset, (for example, dataset.uuid).

Packages which define new storage drivers should generally call add_storage_driver() within their __init__() functions.

source
+endsource
DataSets.newdir — Function
newdir()

Create a new temporary BlobTree which can have files assigned into it and may be assigned to a permanent location in a persistent BlobTree. If not assigned to a permanent location, the temporary tree is cleaned up during garbage collection.

source

Storage Drivers

To add a new kind of data storage backend, implement DataSets.add_storage_driver

DataSets.add_storage_driver — Function
add_storage_driver(driver_name=>storage_opener)

Associate DataSet storage driver named driver_name with storage_opener. When a dataset with storage.driver == driver_name is opened, storage_opener(user_func, storage_config, dataset) will be called. Any existing storage driver registered to driver_name will be overwritten.

As a matter of convention, storage_opener should generally take configuration from storage_config which is just dataset.storage. But to avoid config duplication it may also use the content of dataset, (for example, dataset.uuid).

Packages which define new storage drivers should generally call add_storage_driver() within their __init__() functions.

source
diff --git a/previews/PR71/search/index.html b/previews/PR71/search/index.html index f0e59b2..fbb2978 100644 --- a/previews/PR71/search/index.html +++ b/previews/PR71/search/index.html @@ -1,2 +1,2 @@ -Search · DataSets.jl

Loading search...

    +Search · DataSets.jl

    Loading search...

      diff --git a/previews/PR71/tutorial/index.html b/previews/PR71/tutorial/index.html index 383e455..78e383e 100644 --- a/previews/PR71/tutorial/index.html +++ b/previews/PR71/tutorial/index.html @@ -105,4 +105,4 @@ julia> @datarun main("a_text_file", "a_tree_example"); x = "Hello world!\n" -csv_data = "Name,Age\n\"Aaron\",23\n\"Harry\",42\n"

      In a given program it's possible to have multiple entry points by simply defining multiple @datafunc implementations. In this case @datarun will dispatch to the entry point with the matching DataSet type.

      +csv_data = "Name,Age\n\"Aaron\",23\n\"Harry\",42\n"

      In a given program it's possible to have multiple entry points by simply defining multiple @datafunc implementations. In this case @datarun will dispatch to the entry point with the matching DataSet type.