Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Workflow's nodes' output directories point to TMPDIR instead of base_dir #3542

Open
tsalo opened this issue Feb 1, 2023 · 7 comments · May be fixed by niflows/nipype1-examples#7
Open

Workflow's nodes' output directories point to TMPDIR instead of base_dir #3542

tsalo opened this issue Feb 1, 2023 · 7 comments · May be fixed by niflows/nipype1-examples#7

Comments

@tsalo
Copy link
Member

tsalo commented Feb 1, 2023

Summary

I am trying to write tests for nipype workflows, and I would like to use get_node() to select nodes' outputs, rather than hardcoding paths to output files. However, I've noticed that the nodes' output_dir() methods point to a subfolder in the TMPDIR environment variable, rather than to subfolders in the workflow's base_dir. It seems like this is a bug.

Tagging @mattcieslak as he's been helping me debug this.

Actual behavior

node = wf.get_node("write_string").output_dir() points to temporary directory (/private/var/.../init_minimal_wf/write_string).

Expected behavior

node = wf.get_node("write_string").output_dir() would point to /some/absolute/path/init_minimal_wf/write_string (the node's subfolder in base_dir).

How to replicate the behavior

I wrote a minimal workflow with

"""A test workflow."""
import os

from nipype.interfaces import utility as niu
from nipype.pipeline import engine as pe
from nipype.interfaces.base import (
    BaseInterfaceInputSpec,
    File,
    SimpleInterface,
    TraitedSpec,
    traits,
)


class _WriteStringInputSpec(BaseInterfaceInputSpec):
    in_str = traits.Str(mandatory=True)


class _WriteStringOutputSpec(TraitedSpec):
    out_file = File(exists=True)


class WriteString(SimpleInterface):
    """Write a string to a file."""

    input_spec = _WriteStringInputSpec
    output_spec = _WriteStringOutputSpec

    def _run_interface(self, runtime):
        self._results["out_file"] = os.path.join(runtime.cwd, "out_file.txt")
        with open(self._results["out_file"], "w") as fo:
            fo.write(self.inputs.in_str)

        return runtime


def init_minimal_wf(
    mem_gb=0.1,
    omp_nthreads=1,
    name="init_minimal_wf",
):
    """A minimal workflow to reproduce a bug."""
    workflow = pe.Workflow(name=name)

    inputnode = pe.Node(
        niu.IdentityInterface(fields=["in_str"]),
        name="inputnode",
    )
    outputnode = pe.Node(
        niu.IdentityInterface(fields=["out_file"]),
        name="outputnode",
    )

    write_string = pe.Node(
        WriteString(),
        mem_gb=mem_gb,
        n_procs=omp_nthreads,
        name="write_string",
    )

    # fmt:off
    workflow.connect([
        (inputnode, write_string, [("in_str", "in_str")]),
        (write_string, outputnode, [("out_file", "out_file")]),
    ])
    # fmt:on

    return workflow


if __name__ == "__main__":
    wf = init_minimal_wf()
    wf.inputs.inputnode.in_str = "hello world"
    wf.base_dir = "/some/absolute/path/"  # *not* TMPDIR
    wf.run()
    node = wf.get_node("write_string")
    print(node.output_dir())  # this *should* point to base_dir, but points to TMPDIR instead

Script/Workflow details

Please put URL to code or code here (if not too long).

Platform details:

{'commit_hash': '<not found>',
 'commit_source': '(none found)',
 'networkx_version': '2.8.8',
 'nibabel_version': '3.2.1',
 'nipype_version': '1.8.5',
 'numpy_version': '1.24.1',
 'pkg_path': '/opt/miniconda3/lib/python3.8/site-packages/nipype',
 'scipy_version': '1.8.1',
 'sys_executable': '/opt/miniconda3/bin/python',
 'sys_platform': 'darwin',
 'sys_version': '3.8.5 (default, Sep  4 2020, 02:22:02) \n[Clang 10.0.0 ]',
 'traits_version': '6.3.2'}

Execution environment

My python environment outside container

@effigies
Copy link
Member

@tsalo Sorry for missing this:

res = wf.run()
nodes = {node.fullname: node for node in res.nodes}
nodes['init_minimal_wf.write_string'].output_dir()

@tsalo
Copy link
Member Author

tsalo commented Feb 15, 2023

Thanks @effigies! Does that mean the behavior I was seeing was expected? Or is it a bug that is easier to circumvent than fix?

In any case, I'll try out your recommendation. Do you think this info is worth incorporating into the documentation somewhere? I know most folks don't use pytest to test their nipype workflows, but knowing how to access individual nodes from a run workflow could make it more common.

@effigies
Copy link
Member

Does that mean the behavior I was seeing was expected?

Yes. The workflow graph that you construct is basically a template for the execution graph. The execution graph is flattened from nested workflows to a flat workflow, iterables are expanded, and IdentityInterfaces are removed. The nodes in the execution graph are what are actually run and get updated.

It's definitely worth adding to the docs. If you have a use case, would you be willing to write up a small notebook that we could render?

@tsalo
Copy link
Member Author

tsalo commented Feb 15, 2023

Happy to! I've noticed that a lot of the advanced examples have been offloaded to https://github.com/miykael/nipype_tutorial. Should the notebook go there, or in the nipype docs?

@effigies
Copy link
Member

Neither location has much maintenance effort devoted to it at this point. I have permissions here though, so I guess here is better.

@tsalo
Copy link
Member Author

tsalo commented Feb 15, 2023

Sounds good!

@tsalo
Copy link
Member Author

tsalo commented May 19, 2023

I opened niflows/nipype1-examples#7, but can move it to this repo if necessary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants