Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

For script tasks, tempdir=True with remote executor produces an error #20

Open
spitz-dan-l opened this issue Feb 18, 2022 · 2 comments
Open

Comments

@spitz-dan-l
Copy link
Contributor

Hi there! The following task:

@task()
def main():
    return script('echo HELLO', tempdir=True, executor='batch')

Produces this output:

Traceback (most recent call last):
  File "***/bin/redun", line 11, in <module>
    client.execute()
  File "***/lib/python3.7/site-packages/redun/cli.py", line 1021, in execute
    return args.func(args, extra_args, argv)
  File "***/lib/python3.7/site-packages/redun/cli.py", line 1558, in run_command
    tags=tags,
  File "***/lib/python3.7/site-packages/redun/scheduler.py", line 831, in run
    raise result.value
redun.scripting.ScriptError: Last line: ./.task_command: line 3: cd: /tmp/tmpkt20ugrt.tempdir: No such file or directory

It looks like the design of script() is to create the temp dir when calling script(), but execute the command in a downstream task. Since the downstream task is run remotely, the tempdir won't exist.

@mattrasmus
Copy link
Collaborator

Thanks for submitting this issue.

Usually a tempdir is not needed when running inside a Docker container, since the container provides isolation already. Do you have a case where you really need these options together?

At a minimum we should have a better error message that prevents this combination of options or we should have the tempdir creation occur also within the container.

@spitz-dan-l
Copy link
Contributor Author

Hey there @mattrasmus, thanks for your response!

As you say, the need for tempdir=True is lessened by the fact of running inside a container. In my case, the container I've got happens to have the source directory of a codebase as its working dir. It is possible to overwrite some needed files if you happen to stage in files with the same names.

This could of course be addressed by adjusting the container image to use a different working dir. However, I spent a few minutes writing up a quick PR (#21) showing one way to fix the issue. Happy to update it according to your feedback or simply leave it as a demo.

Cheers!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

2 participants