v0.49.0

What's Changed

• Feat: Prompt for workspace and project in all CLI commands that prompted for WF id by @SebastianMorawiec in #149
• internal: Fix CUDA image publishing workflow failure when no Docker tag is given by @emre-aydin in #152
• Install SSH Client by @emre-aydin in #155
• chore: roll changelog by @alexjuda in #156
• feat: add sdk.get_backend_ids() function. by @BenjaminMummery in #150
• Internal: Fix random failures due to the warning of socket not being closed by @SebastianMorawiec in #161
• Feat: cli easy re login by @BenjaminMummery in #154
• feat: Check tokens before saving in orq by @jamesclark-Zapata in #158
• fix: Tasks that set n_outputs=1 in QE and in-process by @jamesclark-Zapata in #164
• feat: include workspace and project as parameters for listing workflows on CE by @BenjaminMummery in #153
• Feat: add support for Workspaces to all Secret-related functions by @SebastianMorawiec in #157
• feat: give more informative error message when ray cluster isn't running by @BenjaminMummery in #163
• internal: Add a makefile for building the sdk images, tweak for building on M1 by @roryschadler in #159
• Feat: Inherit WS and Project from Studio if used inside portal by @SebastianMorawiec in #162
• feat: Custom Image Support by @jamesclark-Zapata in #166
• fix: Use default image in task resources by @jamesclark-Zapata in #169
• impr: show subprocess error for orq up by @alexjuda in #165
• fix: Bump minimum Pydantic version by @jamesclark-Zapata in #171
• Chore!: Removed WorkflowDef.prepare() and WorkflowRun.start() functions from public API by @SebastianMorawiec in #170
• Docs: Update secrets guide to take workspaces into account by @SebastianMorawiec in #167

New Contributors

• @roryschadler made their first contribution in #159

Full Changelog: v0.48.0...v0.49.0

v0.48.0

What's Changed

• internal: Placeholder GitHub Actions workflow to publish orquestra-sdk-base Docker image by @emre-aydin in #138
• chore: Update changelog for 0.47.0 by @jamesclark-Zapata in #136
• Feat: Add workspace and project parameters to "orq wf submit" CLI command by @SebastianMorawiec in #134
• Feat!: cleanup config api code by @SebastianMorawiec in #130
• Feat: New API functions: list_workspaces and list_projects by @SebastianMorawiec in #139
• chore: pass for spelling, imports, copyright, style, etc. by @BenjaminMummery in #121
• feat: don't return single packed values as sequences by @BenjaminMummery in #141
• fix: Set max retries in Ray by @jamesclark-Zapata in #140
• feat: auto-resolve choice when there is only one. by @BenjaminMummery in #135
• feat: use new login endpoint by @jamesclark-Zapata in #143
• [OP-39] orquestra-sdk-base images for CUDA and non-CUDA workflows by @emre-aydin in #142
• Internal, Fix: rewrite tests to use sockets instead of FS by @SebastianMorawiec in #146
• fix: orq submit with dirty repo by @alexjuda in #147
• internal: Tagging Docker images via optional parameter [OP-39] by @emre-aydin in #145
• feat: support for custom images on CE by @alexjuda in #148
• chore: revert #148 by @alexjuda in #151

Full Changelog: v0.47.0...v0.48.0

v0.47.0

What's Changed

• chore: update changelog for 0.46.0 by @alexjuda in #114
• Feat: workspaceId and projectID support for wf submit by @SebastianMorawiec in #109
• fix: Correct changelog ordering after merge by @jamesclark-Zapata in #116
• Fix: pin polars version used in our tests by @SebastianMorawiec in #118
• Feat: Add new config "self" to submit to CE from within a cluster by @SebastianMorawiec in #117
• fix: Retry getting results from CE after a workflow has succeeded by @jamesclark-Zapata in #115
• fix: Do not modify arg when building Ray workflow by @jamesclark-Zapata in #122
• docs: fix formatting in the Resource Management guide by @alexjuda in #119
• Fix: Fix WorkflowDef.graph function to properly generate graph by @SebastianMorawiec in #124
• chore: add reminder about linking JIRA tickets by @alexjuda in #125
• Feat: Change "self" config name to "auto". "auto" config acts as "local" if not on studio environment by @SebastianMorawiec in #129
• chore!: Merge breaking changes to main by @jamesclark-Zapata in #100
• fix: Custom Image default to None by @jamesclark-Zapata in #131

Full Changelog: v0.46.0...v0.47.0

v0.46.0

What's Changed

• docs: guide for dependency imports by @BenjaminMummery in #74
• feat: raise an exception if workflows have zero tasks. by @BenjaminMummery in #86
• tests: Fix test isolation in docs examples by @jamesclark-Zapata in #87
• fix: Misc QE fixes by @jamesclark-Zapata in #84
• fix: Remove in_process runtime from CLI options by @jamesclark-Zapata in #88
• Feat: Add start time value and sort by it to CLI commands and prompts that list WFs by @SebastianMorawiec in #85
• Fix: Move constant deserialization inside ray remote function by @SebastianMorawiec in #89
• chore: limit github action workflows by @jamesclark-Zapata in #90
• feat: parse binary response into logs. by @BenjaminMummery in #79
• chore: Remove corq by @jamesclark-Zapata in #93
• fix!: multi-output tasks when some outputs are unused by @alexjuda in #75
• refactor: remove obsolete RuntimeInterface methods by @alexjuda in #92
• internal: skip slow tests by @alexjuda in #94
• Revert "fix!: multi-output tasks when some outputs are unused (#75)" by @alexjuda in #97
• Feat!: Remove task.model and task.import_models interfaces by @SebastianMorawiec in #101
• Feat: Allow setting default source imports and dependency imports for all tasks in given WF by @SebastianMorawiec in #102
• feat: Add workflow resources support by @jamesclark-Zapata in #103
• fix: Allow single import as dependency_imports by @jamesclark-Zapata in #104
• Fix: default_dependency_imports as a single import allowed. Fix main reference issue when default_source_import was Inline by @SebastianMorawiec in #106
• docs: guide on workarounds for resource congestion by @BenjaminMummery in #96
• feat: limit CE runs list. by @BenjaminMummery in #111
• feat: print debug information from orq by @alexjuda in #112
• Feat!: Remove env variable that disabled setting resources for local ray by @SebastianMorawiec in #113

Full Changelog: v0.45.1...v0.46.0

v0.45.1

What's Changed

• fix: Ensure decimal types are converted by @jamesclark-Zapata in #82
• chore: Update Changelog for release by @jamesclark-Zapata in #83

Full Changelog: v0.45.0...v0.45.1

v0.45.0

What's Changed

• chore: shift changelog by @alexjuda in #72
• fix: Switch to cloudpickle for pickling by @jamesclark-Zapata in #71
• Chore: remove skipping service integration tests by @SebastianMorawiec in #73
• feat: Scale up CE workflows by @jamesclark-Zapata in #76
• feat: Apply resources to Ray remote functions by @jamesclark-Zapata in #77
• feat: Version metadata added to IR by @jamesclark-Zapata in #78
• docs: All dependencies in quotes by @VolodyaCO in #80
• chore: Update Changelog for release by @jamesclark-Zapata in #81

New Contributors

• @VolodyaCO made their first contribution in #80

Full Changelog: v0.44.0...v0.45.0

v0.44.0

Changelog

🚨 Breaking Changes

• Removed FluentBit-related CLI options: orq {up,down} --fluentbit flag. Logs produced by the local Ray runtime are read directly by the SDK now. This only affects users who used the experimental integration with FluentBit docker container.
• GitImport will no longer be downloaded automatically when using Ray locally. This reverts behavior to v0.42.0.
• Internal configuration environment variables have changed.

🔥 Features

• Secrets can now be used inside workflow functions
• sdk.secrets.get("name") will now use passport-based authorization if ORQUESTRA_PASSPORT_FILE environment variable is set. Otherwise, passing a valid config_name="..." is required.
• Bump Ray version to 2.3
• GithubImport can be used with a username and a secret referring to a "personal access token" to enable private GitHub repositories on Compute Engine. Server side support coming soon!

🐛 Bug Fixes

• Getting full logs produced by Ray workflows. Previously, the dictionary returned by logs_dict = wf_run.get_logs() had just a single entry: {"logs": ["task 1 log", "task 1 log", "task 2 log", "task 2 log"]}. Now, the dictionary has a correct shape: {"task_invocation_id1": ["task 1 log", "task 1 log"], "task_invocation_id2": ["task 2 log", "task 2 log"]}.
• Getting single task logs. Previously orq task logs would raise an unhandled exception. Now, it prints the log lines.
• Workflow run IDs inside logs on CE now match the expected run ID.

💅 Improvements

• orq wf view now shows TaskInvocationIDs instead of TaskRunIDs. This improves usage of orq wf view with other CLI commands that require passing invocation ID, like orq task {logs,results}.
• sdk.WorkflowRun.wait_until_finished() will now print workflow status every now and then.

Internal

• Git URL model changed inside the IR
• orq up will now configure Ray's Plasma directory

PRs

• chore: update changelog after the release by @alexjuda in #45
• docs: describe workaround for Ray & M1 by @alexjuda in #46
• feat: dorq auto login by @jamesclark-Zapata in #41
• internal: remove obsolete iter_logs by @alexjuda in #49
• fix: tag Ray logs with proper task IDs by @alexjuda in #50
• feat: Allow secrets in workflow functions and IR by @jamesclark-Zapata in #52
• feat: IR Secrets in runtimes by @jamesclark-Zapata in #53
• chore: Update ctrl-c handling in orq login by @jamesclark-Zapata in #54
• chore: Pin dill to a release by @jamesclark-Zapata in #55
• docs: remove recipes by @max-radin in #48
• fix: reading single-task logs on Ray by @alexjuda in #51
• refactor: split API module by @alexjuda in #58
• chore: add missing changelog entry by @alexjuda in #57
• feat: use passport auth whenever possible by @alexjuda in #56
• refactor: harden mypy typechecks by @alexjuda in #59
• chore: update Compute Engine error handling by @jamesclark-Zapata in #61
• fix!: remove fluentbit by @alexjuda in #63
• fix: showing task inv IDs by @alexjuda in #60
• fix: Make downloading Gitimports optional in Ray by @jamesclark-Zapata in #62
• chore: Configure Ray plasma storage by @jamesclark-Zapata in #66
• chore: Expand Git URL model by @jamesclark-Zapata in #65
• feat: Private Github repo support by @jamesclark-Zapata in #67
• fix: Ray: Set workflow run ID from environment by @jamesclark-Zapata in #68
• imp: print when polling when awaiting wf completion by @alexjuda in #69
• Feat: bump ray to 2.3 by @SebastianMorawiec in #64
• fix: Make Ray dirs before starting Ray by @jamesclark-Zapata in #70

Full Changelog: v0.43.0...v0.44.0

v0.43.0

What's Changed

🚨 Breaking Changes

• Brand-new orq CLI with simplified command tree and interactive prompts when a required argument isn't passed. New commands:
  • orq workflow submit
  • orq workflow view
  • orq workflow list
  • orq workflow stop
  • orq workflow logs
  • orq workflow results
  • orq wf as a shorthand for orq workflow
  • orq task logs
  • orq task results
  • orq up
  • orq down
  • orq status
  • orq login
• sdk.WorkflowRun.get_logs() doesn't accept any arguments any more. Now, it returns all the logs produced by the tasks in the workflow. If you're interested in only a subset of your workflow's logs, please consider using one of the following filtering options:

from orquestra import sdk
from orquestra.sdk.schema.workflow_run import State

wf_run = sdk.WorkflowRun.by_id("foo")

logs = wf_run.get_logs()
# Option 1
single_task_logs = logs["my_inv_id"]

# Option 2
logs_subset = {id: lines for id, lines in logs.items() if id in ["foo", "bar", "baz"]}

# Option 3
for task in wf_run.get_tasks():
    if task.get_status() == State.FAILED:
        print(task.get_logs())

• sdk.WorkflowRun.get_artifacts() doesn't accept any arguments any more. Now, it returns all the artifacts produced by the tasks in the workflow.
• sdk.TaskRun.get_logs() returns a list of log lines produced by this task. Previously, it returned a dictionary with one entry.
• Executing a workflow on Ray with Git imports will now install them. A known limitation is that this will only work for Git repositories that are Python packages and will fail for Git repositories that are not Python packages.
• The API will no longer accept config_save_file as optional parameters, from now on if you want to use a different config file use the ORQ_CONFIG_PATH environment variable.

🔥 Features

• list_workflow_runs added to the Public API. This lets you list the workflows for a given config, for example sdk.list_workflow_runs("ray") or sdk.list_workflow_runs("prod-d").

🐛 Bug Fixes

• Fixed broken link on docs landing page.
• Internal logs from Ray are no longer displayed.
• Fixed the docstrings for sdk.WorkflowRun.get_artifacts(). It returns a dictionary with TaskInvocationID as keys and whatever the task returns as values.
• Fixed bug where some log line from Ray may be duplicated when viewing logs
• Tasks with duplicate imports will no longer fail when running on QE
• AST parser will no longer print a lot of "Info" messages
• sdk.WorkflowRun.get_logs() now only returns logs produced by the user. Previously, it included internal debug messages produced by Ray.
• Logs from workflows submitted to Ray are now always returned as JSONL lines

New Contributors

• @max-radin made their first contribution in #12
• @BenjaminMummery made their first contribution in #14

Full Changelog: v0.42.0...v0.43.0

v0.42.0

What's Changed

• ci: Add missing makefile by @alexjuda in #2
• Fix: Fix logging scenarios when no config was passed by @SebastianMorawiec in #1

New Contributors

• @alexjuda made their first contribution in #2
• @SebastianMorawiec made their first contribution in #1

Full Changelog: https://github.com/zapatacomputing/orquestra-workflow-sdk/commits/v0.42.0