Merge pull request #41 from scalableminds/volume_tag

Add support for reading and writing <volume> tags

.github/workflows/test.yml

@@ -34,6 +34,6 @@ jobs:
 
     - name: Check Documentation for updates
       run: |
-        poetry run pydoc-markdown -m wknml --render-toc > docs/ci_test.md
+        poetry run pydoc-markdown -m wknml -m wknml.nml_generation -m wknml.nml_utils --render-toc > docs/ci_test.md
         diff docs/ci_test.md docs/wknml.md
         rm docs/ci_test.md

README.md

@@ -69,7 +69,7 @@ poetry version <patch, minor, major>
 
 If necessary, rebuild the documentation and commit to repository:
 ```
-poetry run pydoc-markdown -m wknml --render-toc > docs/wknml.md
+poetry run pydoc-markdown -m wknml -m wknml.nml_generation -m wknml.nml_utils --render-toc > docs/wknml.md
 ```
 
 # License

docs/wknml.md

@@ -8,9 +8,20 @@
   * [Branchpoint](#wknml.Branchpoint)
   * [Group](#wknml.Group)
   * [Comment](#wknml.Comment)
+  * [Volume](#wknml.Volume)
   * [NML](#wknml.NML)
   * [parse\_nml](#wknml.parse_nml)
   * [write\_nml](#wknml.write_nml)
+* [wknml.nml\_generation](#wknml.nml_generation)
+  * [random\_color\_rgba](#wknml.nml_generation.random_color_rgba)
+  * [discard\_children\_hierarchy](#wknml.nml_generation.discard_children_hierarchy)
+  * [globalize\_tree\_ids](#wknml.nml_generation.globalize_tree_ids)
+  * [globalize\_node\_ids](#wknml.nml_generation.globalize_node_ids)
+  * [generate\_nml](#wknml.nml_generation.generate_nml)
+  * [generate\_graph](#wknml.nml_generation.generate_graph)
+  * [nml\_tree\_to\_graph](#wknml.nml_generation.nml_tree_to_graph)
+  * [extract\_nodes\_and\_edges\_from\_graph](#wknml.nml_generation.extract_nodes_and_edges_from_graph)
+* [wknml.nml\_utils](#wknml.nml_utils)
 
 <a name="wknml"></a>
 # wknml
@@ -24,17 +35,22 @@ class NMLParameters(NamedTuple)
 
 Contains common metadata for NML files
 
+**Notes**:
+
+  Setting a task bounding boxes will cause wK to 1) render these visually and 2) prevent data loading from outside them.
+
+
 **Attributes**:
 
-- `name` - str
-- `scale` - Vector3
-- `offset` - Optional[Vector3]
-- `time` - Optional[int]
-- `editPosition` - Optional[Vector3]
-- `editRotation` - Optional[Vector3]
-- `zoomLevel` - Optional[float]
-- `taskBoundingBox` - Optional[IntVector6]
-- `userBoundingBox` - Optional[IntVector6]
+- `name` _str_ - Name of a dataset that the annotation is based on. Will cause wK to open the given skeleton annotation with the referenced dataset.
+- `scale` _Vector3_ - Voxel scale of the referenced dataset in nanometers.
+- `offset` _Optional[Vector3]_ - Deprecated. Kept for backward compatibility.
+- `time` _Optional[int]_ - A UNIX timestamp marking the creation time & date of an annotation.
+- `editPosition` _Optional[Vector3]_ - The position of the wK camera when creating/downloading an annotation
+- `editRotation` _Optional[Vector3]_ - The rotation of the wK camera when creating/downloading an annotation
+- `zoomLevel` _Optional[float]_ - The zoomLevel of the wK camera when creating/downloading an annotation
+- `taskBoundingBox` _Optional[IntVector6]_ - A custom bounding box specified as part of a [wK task](https://docs.webknossos.org/guides/tasks). Will be rendered in wK.
+- `userBoundingBox` _Optional[IntVector6]_ - A custom user-defined bounding box. Will be rendered in wK.
 
 <a name="wknml.Node"></a>
 ## Node Objects
@@ -47,15 +63,15 @@ A webKnossos skeleton node annotation object.
 
 **Attributes**:
 
-- `id` - int
-- `position` - Vector3
-- `radius` - Optional[float]
-- `rotation` - Optional[Vector3]
-- `inVp` - Optional[int]
-- `inMag` - Optional[int]
-- `bitDepth` - Optional[int]
-- `interpolation` - Optional[bool]
-- `time` - Optional[int]
+- `id` _int_ - A unique identifier
+- `position` _Vector3_ - 3D position of a node. Format: [x, y, z]
+- `radius` _Optional[float]_ - Radius of a node when rendered in wK. Unit: nanometers (nm)
+- `rotation` _Optional[Vector3]_ - 3D rotation of the camera when the node was annotated. Mostly relevant for `Flight` mode to resume in the same direction when returning to `Flight` mode.
+- `inVp` _Optional[int]_ - Enumeration of the wK UI viewport in which the node was annotated. `0`: XY plane, `1`: YZ plane. `2`: XY plane, `3`: 3D viewport
+- `inMag` _Optional[int]_ - wK rendering magnification-level when the node was annotated. Lower magnification levels typically indicate a "zoomed-in" workflow resulting in more accurate annotations.
+- `bitDepth` _Optional[int]_ - wK rendering bit-depth when the node was annotated. 4bit (lower data quality) or 8bit (regular quality). Lower quality data rendering might lead to less accurate annotations.
+- `interpolation` _Optional[bool]_ - wK rendering interpolation flag when the node was annotated. Interpolated data rendering might lead to less accurate annotations.
+- `time` _Optional[int]_ - A Unix timestamp
 
 <a name="wknml.Edge"></a>
 ## Edge Objects
@@ -100,7 +116,7 @@ A webKnossos branchpoint, i.e. a skeleton node with more than one outgoing edge.
 
 **Attributes**:
 
-- `id` _int_ - node id reference
+- `id` _int_ - Reference to a `Node` ID
 - `time` _int_ - Unix timestamp
 
 <a name="wknml.Group"></a>
@@ -114,9 +130,9 @@ A container to group several skeletons (trees) together. Mostly for cosmetic or
 
 **Attributes**:
 
-- `id` - int
-- `name` - str
-- `children` - List[Group]
+- `id` _int_ - A u unique group identifier
+- `name` _str_ - NameA  of the group. Will be displayed in wK UI
+- `children` _List[Group]_ - List of all sub-groups belonging to this parent element for nested structures
 
 <a name="wknml.Comment"></a>
 ## Comment Objects
@@ -129,8 +145,23 @@ A single comment belonging to a skeleton node.
 
 **Attributes**:
 
-- `node` _int_ - node id reference
-- `content` _str_ - supports Markdown
+- `node` _int_ - Reference to a `Node` ID
+- `content` _str_ - A free text field. Supports Markdown formatting.
+
+<a name="wknml.Volume"></a>
+## Volume Objects
+
+```python
+class Volume(NamedTuple)
+```
+
+A metadata reference to a wK volume annotation. Typically, the volum annotation data is provided a ZIP file in the same directory as the skeleton annotation.
+
+**Attributes**:
+
+- `id` _int_ - A unique Identifier
+- `location` _str_ - A path to a ZIP file containing a wK volume annotation
+- `fallback_layer` _Optional[str]_ - name of an already existing wK volume annotation segmentation layer (aka "fallback layer")
 
 <a name="wknml.NML"></a>
 ## NML Objects
@@ -143,11 +174,12 @@ A complete webKnossos skeleton annotation object contain one or more skeletons (
 
 **Attributes**:
 
-- `parameters` - NMLParameters
-- `trees` - List[Tree]
-- `branchpoints` - List[Branchpoint]
-- `comments` - List[Comment]
-- `groups` - List[Group]
+- `parameters` _NMLParameters_ - All the metadata attributes associated with a wK skeleton annotation.
+- `trees` _List[Tree]_ - A list of all skeleton/tree objects. Usually contains of the information.
+- `branchpoints` _List[Branchpoint]_ - A list of all branchpoint objects.
+- `comments` _List[Comment]_ - A list of all comment objects.
+- `groups` _List[Group]_ - A list of all group objects.
+- `volume` _Optional[Volume]_ - A reference to any volume data that might reside in the directory as the NML file.
 
 <a name="wknml.parse_nml"></a>
 #### parse\_nml
@@ -158,7 +190,7 @@ parse_nml(file: BinaryIO) -> NML
 
 Reads a webKnossos NML skeleton file from disk, parses it and returns an NML Python object
 
-**Attributes**:
+**Arguments**:
 
 - `file` _BinaryIO_ - A Python file handle
 
@@ -197,3 +229,119 @@ Writes an NML object to a file on disk.
   wknml.write_nml(f, nml)
   ```
 
+<a name="wknml.nml_generation"></a>
+# wknml.nml\_generation
+
+<a name="wknml.nml_generation.random_color_rgba"></a>
+#### random\_color\_rgba
+
+```python
+random_color_rgba() -> Tuple[float, float, float, float]
+```
+
+A utility to generate a new random RGBA color.
+
+<a name="wknml.nml_generation.discard_children_hierarchy"></a>
+#### discard\_children\_hierarchy
+
+```python
+discard_children_hierarchy(groups: List[Group]) -> List[Group]
+```
+
+A utility to flatten the group structure. All sub-groups will become top-level items.
+
+<a name="wknml.nml_generation.globalize_tree_ids"></a>
+#### globalize\_tree\_ids
+
+```python
+globalize_tree_ids(group_dict: Dict[str, List[nx.Graph]])
+```
+
+A utility to in-place re-assign new and globally unqiue IDs to all Tree objects. Starts with ID 1
+
+**Arguments**:
+
+- `group_dict` _Dict[str, List[nx.Graph]]_ - A mapping of group names to a list of tree as NetworkX graph objects
+
+<a name="wknml.nml_generation.globalize_node_ids"></a>
+#### globalize\_node\_ids
+
+```python
+globalize_node_ids(group_dict: Dict[str, List[nx.Graph]])
+```
+
+A utility to in-place re-assign new and globally unqiue IDs to all Node objects. Edges are updated accordingly. Starts with ID 1.
+
+Note: Does not update any `Comment`s or `BranchPoint`s referencing these nodes.
+
+**Arguments**:
+
+- `group_dict` _Dict[str, List[nx.Graph]]_ - A mapping of group names to a list of tree as NetworkX graph objects
+
+<a name="wknml.nml_generation.generate_nml"></a>
+#### generate\_nml
+
+```python
+generate_nml(tree_dict: Union[List[nx.Graph], Dict[str, List[nx.Graph]]], parameters: Dict[str, Any] = {}, globalize_ids: bool = True, volume: Optional[Dict[str, Any]] = None) -> NML
+```
+
+A utility to convert a [NetworkX graph object](https://networkx.org/) into wK NML skeleton annotation object. Accepts both a simple list of multiple skeletons/trees or a dictionary grouping skeleton inputs.
+
+**Arguments**:
+
+- `tree_dict` _Union[List[nx.Graph], Dict[str, List[nx.Graph]]]_ - A list of wK tree-like structures as NetworkX graphs or a dictionary of group names and same list of NetworkX tree objects.
+- `parameters` _Dict[str, Any]_ - A dictionary representation of the skeleton annotation metadata. See `NMLParameters` for accepted attributes.
+- `globalize_ids` _bool = True_ - An option to re-assign new, globally unique IDs to all skeletons. Default: `True`
+- `volume` _Optional[Dict[str, Any]] = None_ - A dictionary representation of a reference to wK a volume annotation. See `Volume` object for attributes.
+
+
+**Returns**:
+
+- `nml` _NML_ - A wK NML skeleton annotation object
+  1. A dictionary with group names as keys and lists of all respective NML trees as values
+  2. A dictionary representation of the NML metadata parameters
+
+<a name="wknml.nml_generation.generate_graph"></a>
+#### generate\_graph
+
+```python
+generate_graph(nml: NML) -> Tuple[Dict[str, List[nx.Graph]], Dict[Text, any]]
+```
+
+A utility to convert a wK NML object into a [NetworkX graph object](https://networkx.org/). Skeletons/Trees are grouped by the provided groups in the NML file.
+
+**Arguments**:
+
+- `nml` _NML_ - A wK NML skeleton annotation object
+
+
+**Returns**:
+
+  A tuple consisting of:
+  1. A dictionary with group names as keys and lists of all respective NML trees as values
+  2. A dictionary representation of the NML metadata parameters. See `NMLParameters` for attributes.
+
+<a name="wknml.nml_generation.nml_tree_to_graph"></a>
+#### nml\_tree\_to\_graph
+
+```python
+nml_tree_to_graph(tree: Tree) -> nx.Graph
+```
+
+A utility to convert a single wK Tree object into a [NetworkX graph object](https://networkx.org/).
+
+<a name="wknml.nml_generation.extract_nodes_and_edges_from_graph"></a>
+#### extract\_nodes\_and\_edges\_from\_graph
+
+```python
+extract_nodes_and_edges_from_graph(graph: nx.Graph) -> Tuple[List[Node], List[Edge]]
+```
+
+A utility to convert a single [NetworkX graph object](https://networkx.org/) into a list of `Node` objects and `Edge` objects.
+
+Return
+    Tuple[List[Node], List[Edge]]: A tuple containing both all nodes and all edges
+
+<a name="wknml.nml_utils"></a>
+# wknml.nml\_utils
+

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "wknml"
-version = "1.0.2"
+version = "1.0.3"
 description = "Python package to work with webKnossos NML skeleton files"
 authors = ["scalable minds <[email protected]>"]
 readme = "README.md"

testdata/dataset.fixture.nml

@@ -105,4 +105,5 @@
   </comments>
   <groups>
   </groups>
+  <volume id="1" location="data.zip" />
 </things>

testdata/dataset.nml.snapshot

@@ -99,4 +99,5 @@
   </branchpoints>
   <comments />
   <groups />
+  <volume id="1" location="data.zip" />
 </things>

tests/test_nml_objects.py

@@ -9,6 +9,7 @@
     Tree,
     Branchpoint,
     NMLParameters,
+    Volume,
 )
 from networkx.classes.graph import Graph
 
@@ -66,39 +67,52 @@ def test_build_new_nml_onject():
         userBoundingBox=[1, 2, 3, 1, 2, 3],
     )
 
+    volume = Volume(
+        id=next(id_counter),
+        location="some/path/to/data.zip",
+        fallback_layer="segmentation_layer",
+    )
+
     nml = NML(
         parameters=parameters,
         trees=[tree],
         branchpoints=[branchpoint],
         comments=[comment],
         groups=[group3],
+        volume=volume,
     )
 
     # pass test if all objects can be constructed successfully
     assert True
 
 
 def test_optional_parameters():
-    """Test minimal object constructor. Are the defaults applied."""
+    """Test minimal object constructor. Check to see if the defaults are applied."""
     id_counter = count()
 
     node = Node(id=next(id_counter), position=[1, 2, 3])
     assert node.radius == None
-    assert node.rotation == None
-    assert node.inVp == None
-    assert node.inMag == None
-    assert node.bitDepth == None
-    assert node.interpolation == None
-    assert node.time == None
+    assert node.rotation is None
+    assert node.inVp is None
+    assert node.inMag is None
+    assert node.bitDepth is None
+    assert node.interpolation is None
+    assert node.time is None
 
     tree = Tree(next(id_counter), edges=[], nodes=[], name="Test", color=[])
-    assert tree.groupId == None
+    assert tree.groupId is None
 
     parameters = NMLParameters("Test Annotation", [1, 2, 3])
-    assert parameters.offset == None
-    assert parameters.time == None
-    assert parameters.editPosition == None
-    assert parameters.editRotation == None
-    assert parameters.zoomLevel == None
-    assert parameters.taskBoundingBox == None
-    assert parameters.userBoundingBox == None
+    assert parameters.offset is None
+    assert parameters.time is None
+    assert parameters.editPosition is None
+    assert parameters.editRotation is None
+    assert parameters.zoomLevel is None
+    assert parameters.taskBoundingBox is None
+    assert parameters.userBoundingBox is None
+
+    volume = Volume(
+        id=next(id_counter),
+        location="some/path/to/data.zip",
+    )
+    assert volume.fallback_layer is None