Skip to content

Latest commit

 

History

History
490 lines (377 loc) · 29.7 KB

RELEASE_NOTES.md

File metadata and controls

490 lines (377 loc) · 29.7 KB
Raw

0.5.6

    )                    *      (     
 ( /(                  (  `     )\ )  
 )\())    (            )\))(   (()/(  
((_)\    ))\    (     ((_)()\   /(_)) 
 _((_)  /((_)   )\ )  (_()((_) (_))   
|_  /  (_))    _(_/(  |  \/  | | |    
 / /   / -_)  | ' \)) | |\/| | | |__  
/___|  \___|  |_||_|  |_|  |_| |____|

This release fixes some known bugs from previous releases and especially 0.5.5. Therefore, upgrading to 0.5.6 is a breaking change. You must do the following in order to proceed with this version:

cd zenml_enabled_repo
rm -rf .zen/

And then start again with ZenML init:

pip install --upgrade zenml
zenml init

New Features

  • Added zenml example run [EXAMPLE_RUN_NAME] feature: The ability to run an example with one command. In order to run this, do zenml example pull first and see all examples available by running zenml example list.
  • Added ability to specify a .dockerignore file before running pipelines on Kubeflow.
  • Kubeflow Orchestrator is now leaner and faster.
  • Added the describe command group to the CLI for groups stack, orchestrator, artifact-store, and metadata-store. E.g. zenml stack describe

Bug fixes and minor improvements

  • Adding StepContext to a branch now invalidates caching by default. Disable explicitly with enable_cache=True.
  • Docs updated to reflect minor changes in CLI commands.
  • CLI list commands now mentions active component. Try zenml stack list to check this out.
  • zenml version now has cooler art.

What's Changed

  • Delete blog reference from release notes by @alex-zenml in zenml-io#228
  • Docs updates by @alex-zenml in zenml-io#229
  • Update kubeflow guide by @schustmi in zenml-io#230
  • Updated quickstart to reflect newest zenml version by @alexej-zenml in zenml-io#231
  • Add KFP GCP example readme by @schustmi in zenml-io#233
  • Baris/update docs with class api by @bcdurak in zenml-io#232
  • fixing a small typo [ci skip] by @bcdurak in zenml-io#236
  • Hamza/docs last min updates by @htahir1 in zenml-io#234
  • fix broken links by @alex-zenml in zenml-io#237
  • added one more page for standardized artifacts [ci skip] by @bcdurak in zenml-io#238
  • Unified use of cli_utils.print_table for all table format cli printouts by @AlexejPenner in zenml-io#240
  • Remove unused tfx kubeflow code by @schustmi in zenml-io#239
  • Relaxed typing requirements for cli_utils.print_table by @AlexejPenner in zenml-io#241
  • Pass input artifact types to kubeflow container entrypoint by @schustmi in zenml-io#242
  • Catch duplicate run name error and throw custom exception by @schustmi in zenml-io#243
  • Improved logs by @htahir1 in zenml-io#244
  • CLI active component highlighting by @alex-zenml in zenml-io#245
  • Baris/eng 244 clean up by @bcdurak in zenml-io#246
  • CLI describe command by @alex-zenml in zenml-io#248
  • Alexej/eng 35 run examples from cli by @AlexejPenner in zenml-io#253
  • CLI argument and option flag consistency improvements by @alex-zenml in zenml-io#250
  • Invalidate caching when a step requires a step context by @schustmi in zenml-io#252
  • Implement better error messages for custom step output artifact types by @schustmi in zenml-io#254
  • Small improvements by @schustmi in zenml-io#251
  • Kubeflow dockerignore by @schustmi in zenml-io#249
  • Rename container registry folder to be consistent with the other stack components by @schustmi in zenml-io#257
  • Update todo script by @schustmi in zenml-io#256
  • Update docs following CLI change by @alex-zenml in zenml-io#255
  • Bump mypy version by @schustmi in zenml-io#258
  • Kubeflow Windows daemon alternative by @schustmi in zenml-io#259
  • Run pre commit in local environment by @schustmi in zenml-io#260
  • Hamza/eng 269 move beam out by @htahir1 in zenml-io#262
  • Update docs by @alex-zenml in zenml-io#261
  • Hamza/update readme with contribitions by @htahir1 in zenml-io#271
  • Hamza/eng 256 backoff analytics by @htahir1 in zenml-io#270
  • Add spellcheck by @alex-zenml in zenml-io#264
  • Using the pipeline run name to explicitly access when explaining the … by @AlexejPenner in zenml-io#263
  • Import user main module in kubeflow entrypoint to make sure all components are registered by @schustmi in zenml-io#273
  • Fix cli version command by @schustmi in zenml-io#272
  • User is informed of version mismatch and example pull defaults to cod… by @AlexejPenner in zenml-io#274
  • Hamza/eng 274 telemetry by @htahir1 in zenml-io#275
  • Update docs with right commands and events by @htahir1 in zenml-io#276
  • Fixed type annotation for some python versions by @AlexejPenner in zenml-io#277

Full Changelog: https://github.com/zenml-io/zenml/compare/0.5.5...0.5.6

0.5.5

ZenML 0.5.5 is jam-packed with new features to take your ML pipelines to the next level. Our three biggest new features: Kubeflow Pipelines, CLI support for our integrations and Standard Interfaces. That’s right, Standard Interfaces are back!

What's Changed

New Contributors

  • @alexej-zenml made their first contribution in zenml-io#223

Full Changelog: https://github.com/zenml-io/zenml/compare/0.5.4...0.5.5

0.5.4

0.5.4 adds a lineage tracking integration to visualize lineage of pipeline runs! It also includes numerous bug fixes and optimizations.

What's Changed

  • Fix typos by @alex-zenml in zenml-io#192
  • Fix Apache Beam bug by @alex-zenml in zenml-io#194
  • Fix apache beam logging bug by @alex-zenml in zenml-io#195
  • Add step context by @schustmi in zenml-io#196
  • Init docstrings by @alex-zenml in zenml-io#197
  • Hamza/small fixes by @htahir1 in zenml-io#199
  • Fix writing to metadata store with airflow orchestrator by @schustmi in zenml-io#198
  • Use pipeline parameter name as step name in post execution by @schustmi in zenml-io#200
  • Add error message when step name is not in metadata store by @schustmi in zenml-io#201
  • Add option to set repo location using an environment variable by @schustmi in zenml-io#202
  • Run cloudbuild after pypi publish by @schustmi in zenml-io#203
  • Refactor component generation by @schustmi in zenml-io#204
  • Removed unnecessary panel dependency by @htahir1 in zenml-io#206
  • Updated README to successively install requirements by @AlexejPenner in zenml-io#205
  • Store active stack in local config by @schustmi in zenml-io#208
  • Hamza/eng 125 lineage tracking vis by @htahir1 in zenml-io#207

New Contributors

  • @AlexejPenner made their first contribution in zenml-io#205

Full Changelog: https://github.com/zenml-io/zenml/compare/0.5.3...0.5.4

0.5.3

Version 0.5.3 adds statistics visualizations, greatly improved speed for CLI commands as well as lots of small improvements to the pipeline and step interface.

What's Changed

  • Make tests run in a random order by @alex-zenml in zenml-io#160
  • Connect steps using *args by @schustmi in zenml-io#162
  • Move location of repobeats image by @alex-zenml in zenml-io#163
  • Hamza/add sam by @htahir1 in zenml-io#165
  • Pipeline initialization with *args by @schustmi in zenml-io#164
  • Improve detection of third party modules during class resolving by @schustmi in zenml-io#167
  • Merge path_utils into fileio & refactor what was left by @alex-zenml in zenml-io#168
  • Update docker files by @schustmi in zenml-io#169
  • Hamza/deploy api reference by @htahir1 in zenml-io#171
  • API Reference by @schustmi in zenml-io#172
  • Add color back into our github actions by @alex-zenml in zenml-io#176
  • Refactor tests not raising by @alex-zenml in zenml-io#177
  • Improve step and pipeline interface by @schustmi in zenml-io#175
  • Alex/eng 27 windows bug again by @htahir1 in zenml-io#178
  • Automated todo tracking by @schustmi in zenml-io#173
  • Fix mypy issues related to windows by @schustmi in zenml-io#179
  • Include Github URL to TODO comment in issue by @schustmi in zenml-io#181
  • Create Visualizers logic by @htahir1 in zenml-io#182
  • Add README for visualizers examples by @alex-zenml in zenml-io#184
  • Allow None as default value for BaseStep configs by @schustmi in zenml-io#185
  • Baris/eng 37 standard import check by @bcdurak in zenml-io#183
  • Replace duplicated code by call to source_utils.resolve_class by @schustmi in zenml-io#186
  • Remove unused base enum cases by @schustmi in zenml-io#187
  • Testing mocks for CLI examples command by @alex-zenml in zenml-io#180
  • Set the correct module for steps created using our decorator by @schustmi in zenml-io#188
  • Fix some cli commands by @schustmi in zenml-io#189
  • Tag jira issues for which the todo was deleted by @schustmi in zenml-io#190
  • Remove deadlinks by @alex-zenml in zenml-io#191

Full Changelog: https://github.com/zenml-io/zenml/compare/0.5.2...0.5.3

0.5.2

0.5.2 brings an improved post-execution workflow and lots of minor changes and upgrades for the developer experience when creating pipelines. It also improves the Airflow orchestrator logic to accommodate for more real world scenarios. Check out the low level API guide for more details!

What's Changed

  • Fix autocomplete for step and pipeline decorated functions by @schustmi in zenml-io#144
  • Add reference docs for CLI example functionality by @alex-zenml in zenml-io#145
  • Fix mypy integration by @schustmi in zenml-io#147
  • Improve Post-Execution Workflow by @schustmi in zenml-io#146
  • Fix CLI examples bug by @alex-zenml in zenml-io#148
  • Update quickstart example notebook by @alex-zenml in zenml-io#150
  • Add documentation images by @alex-zenml in zenml-io#151
  • Add prettierignore to gitignore by @alex-zenml in zenml-io#154
  • Airflow orchestrator improvements by @schustmi in zenml-io#153
  • Google colab added by @htahir1 in zenml-io#155
  • Tests for core and cli modules by @alex-zenml in zenml-io#149
  • Add Paperspace environment check by @alex-zenml in zenml-io#156
  • Step caching by @schustmi in zenml-io#157
  • Add documentation for pipeline step parameter and run name configuration by @schustmi in zenml-io#158
  • Automatically disable caching if the step function code has changed by @schustmi in zenml-io#159

Full Changelog: https://github.com/zenml-io/zenml/compare/0.5.1...0.5.2

0.5.1

0.5.1 builds on top of Slack of the 0.5.0 release with quick bug updates.

Overview

  • Pipeline can now be run via a YAML file. #132
  • CLI now let's you pull directly from GitHub examples folder. 🔥 Amazing @alex-zenml with #141!
  • ZenML now has full mypy compliance. 🎉 Thanks @schustmi for #140!
  • Numerous bugs and performance improvements. #136, @bcdurak great job with #142
  • Added new docs with a low level API guide. #143

Our roadmap goes into further detail on the timeline. Vote on the next features now.

We encourage every user (old or new) to start afresh with this release. Please go over our latest docs and examples to get a hang of the new system.

0.5.0

This long-awaited ZenML release marks a seminal moment in the project's history. We present to you a complete revamp of the internals of ZenML, with a fresh new design and API. While these changes are significant, and have been months in the making, the original vision of ZenML has not wavered. We hope that the ZenML community finds the new design choices easier to grasp and use, and we welcome feedback on the issues board.

Warning

0.5.0 is a complete API change from the previous versions of ZenML, and is a breaking upgrade. Fundamental concepts have been changed, and therefore backwards compatibility is not maintained. Please use only this version with fresh projects.

With such significant changes, we expect this release to also be breaking. Please report any bugs in the issue board, and they should be addressed in upcoming releases.

Overview

  • Introducing a new functional API for creating pipelines and steps. This is now the default mechanism for building ZenML pipelines. read more
  • Steps now use Materializers to handle artifact serialization/deserialization between steps. This is a powerful change, and will be expanded upon in the future. read more
  • Introducing the new Stack paradigm: Easily transition from one MLOps stack to the next with a few CLI commands read more
  • Introducing a new Artifact, Typing, and Annotation system, with pydantic (and dataclasses) support read more
  • Deprecating the pipelines_dir: Now individual pipelines will be stored in their metadata stores, making the metadata store a single source of truth. read more
  • Deprecating the YAML config file: ZenML no longer natively compiles to an intermediate YAML-based representation. Instead, it compiles and deploys directly into the selected orchestrator's representation. While we do plan to support running pipelines directly through YAML in the future, it will no longer be the default route through which pipelines are run. read more about orchestrators here

Technical Improvements

  • A completely new system design, please refer to the docs.
  • Better type hints and docstrings.
  • Auto-completion support.
  • Numerous performance improvements and bug fixes, including a smaller dependency footprint.

What to expect in the next weeks and the new ZenML

Currently, this release is bare bones. We are missing some basic features which used to be part of ZenML 0.3.8 (the previous release):

  • Standard interfaces for TrainingPipeline.
  • Individual step interfaces like PreprocesserStep, TrainerStep, DeployerStep etc. need to be rewritten from within the new paradigm. They should be included in the non-RC version of this release.
  • A proper production setup with an orchestrator like Airflow.
  • A post-execution workflow to analyze and inspect pipeline runs.
  • The concept of Backends will evolve into a simple mechanism of transitioning individual steps into different runners.
  • Support for KubernetesOrchestrator, KubeflowOrchestrator, GCPOrchestrator and AWSOrchestrator are also planned.
  • Dependency management including Docker support is planned.

Our roadmap goes into further detail on the timeline.

We encourage every user (old or new) to start afresh with this release. Please go over our latest docs and examples to get a hang of the new system.

Onwards and upwards to 1.0.0!

0.5.0rc2

This long-awaited ZenML release marks a seminal moment in the project's history. We present to you a complete revamp of the internals of ZenML, with a fresh new design and API. While these changes are significant, and have been months in the making, the original vision of ZenML has not wavered. We hope that the ZenML community finds the new design choices easier to grasp and use, and we welcome feedback on the issues board.

Warning

0.5.0rc0 is a complete API change from the previous versions of ZenML, and is a breaking upgrade. Fundamental concepts have been changed, and therefore backwards compatibility is not maintained. Please use only this version with fresh projects.

With such significant changes, we expect this release to also be breaking. Please report any bugs in the issue board, and they should be addressed in upcoming releases.

Overview

  • Introducing a new functional API for creating pipelines and steps. This is now the default mechanism for building ZenML pipelines. read more
  • Introducing the new Stack paradigm: Easily transition from one MLOps stack to the next with a few CLI commands read more
  • Introducing a new Artifact, Typing, and Annotation system, with pydantic (and dataclasses) support read more
  • Deprecating the pipelines_dir: Now individual pipelines will be stored in their metadata stores, making the metadata store a single source of truth. read more
  • Deprecating the YAML config file: ZenML no longer natively compiles to an intermediate YAML-based representation. Instead, it compiles and deploys directly into the selected orchestrator's representation. While we do plan to support running pipelines directly through YAML in the future, it will no longer be the default route through which pipelines are run. read more about orchestrators here

Technical Improvements

  • A completely new system design, please refer to the docs.
  • Better type hints and docstrings.
  • Auto-completion support.
  • Numerous performance improvements and bug fixes, including a smaller dependency footprint.

What to expect in the next weeks and the new ZenML

Currently, this release is bare bones. We are missing some basic features which used to be part of ZenML 0.3.8 (the previous release):

  • Standard interfaces for TrainingPipeline.
  • Individual step interfaces like PreprocesserStep, TrainerStep, DeployerStep etc. need to be rewritten from within the new paradigm. They should be included in the non-RC version of this release.
  • A proper production setup with an orchestrator like Airflow.
  • A post-execution workflow to analyze and inspect pipeline runs.
  • The concept of Backends will evolve into a simple mechanism of transitioning individual steps into different runners.
  • Support for KubernetesOrchestrator, KubeflowOrchestrator, GCPOrchestrator and AWSOrchestrator are also planned.
  • Dependency management including Docker support is planned.

Our roadmap goes into further detail on the timeline.

We encourage every user (old or new) to start afresh with this release. Please go over our latest docs and examples to get a hang of the new system.

Onwards and upwards to 1.0.0!

0.3.7.1

This release fixes some known bugs from previous releases and especially 0.3.7. Same procedure as always, please delete existing pipelines, metadata, and artifact stores.

cd zenml_enabled_repo
rm -rf pipelines/
rm -rf .zenml/

And then another ZenML init:

pip install --upgrade zenml
cd zenml_enabled_repo
zenml init

New Features

  • Introduced new zenml example CLI sub-group: Easily pull examples via zenml to check it out.
zenml example pull # pulls all examples in `zenml_examples` directory
zenml example pull EXAMPLE_NAME  # pulls specific example
zenml example info EXAMPLE_NAME  # gives quick info regarding example

Thanks Michael Xu for the suggestion!

  • Updated examples with new zenml examples paradigm for examples.

Bug Fixes + Refactor

  • ZenML now works on Windows -> Thank you @Franky007Bond for the heads up.
  • Updated numerous bugs in examples directory. Also updated README's.
  • Fixed remote orchestration logic -> Now remote orchestration works.
  • Changed datasource to_config to include reference to backend, metadata, and artifact store.

0.3.7

0.3.7 is a much-needed, long-awaited, big refactor of the Datasources paradigm of ZenML. There are also bug fixes, improvements, and more!

For those upgrading from an older version of ZenML, we ask to please delete their old pipelines dir and .zenml folders and start afresh with a zenml init.

If only working locally, this is as simple as:

cd zenml_enabled_repo
rm -rf pipelines/
rm -rf .zenml/

And then another ZenML init:

pip install --upgrade zenml
cd zenml_enabled_repo
zenml init

New Features

  • The inner-workings of the BaseDatasource have been modified along with the concrete implementations. Now, there is no relation between a DataStep and a Datasource: A Datasource holds all the logic to version and track itself via the new commit paradigm.

  • Introduced a new interface for datasources, the process method which is responsible for ingesting data and writing to TFRecords to be consumed by later steps.

  • Datasource versions (snapshots) can be accessed directly via the commits paradigm: Every commit is a new version of data.

  • Added JSONDatasource and TFRecordsDatasource.

Bug Fixes + Refactor

A big thanks to our new contributor @aak7912 for the help in this release with issue #71 and PR #75.

  • Added an example for regression.
  • compare_training_runs() now takes an optional datasource parameter to filter by datasource.
  • Trainer interface refined to focus on run_fn rather than other helper functions.
  • New docs released with a streamlined vision and coherent storyline: https://docs.zenml.io
  • Got rid of unnecessary Torch dependency with base ZenML version.

0.3.6

0.3.6 is a more inwards-facing release as part of a bigger effort to create a more flexible ZenML. As a first step, ZenML now supports arbitrary splits for all components natively, freeing us from the train/eval split paradigm. Here is an overview of changes:

New Features

  • The inner-workings of the BaseTrainerStep, BaseEvaluatorStep and the BasePreprocesserStep have been modified along with their respective components to work with the new split_mapping. Now, users can define arbitrary splits (not just train/eval). E.g. Doing a train/eval/test split is possible.

  • Within the instance of a TrainerStep, the user has access to input_patterns and output_patterns which provide the required uris with respect to their splits for the input and output(test_results) examples.

  • The built-in trainers are modified to work with the new changes.

Bug Fixes + Refactor

A big thanks to our new super supporter @zyfzjsc988 for most of the feedback that led to bug fixes and enhancements for this release:

  • #63: Now one can specify which ports ZenML opens its add-on applications.
  • #64 Now there is a way to list integrations with the following code:
from zenml.utils.requirements_utils import list_integrations.
list_integrations()
  • Fixed #61: view_anomalies() breaking in the quickstart.
  • Analytics is now opt-in by default, to get rid of the unnecessary prompt at zenml init. Users can still freely opt-out by using the CLI:
zenml config analytics opt-out

Again, the telemetry data is fully anonymized and just used to improve the product. Read more here

0.3.5

New Features

  • Added a new interface into the trainer step called test_fn which is utilized to produce model predictions and save them as test results

  • Implemented a new evaluator step called AgnosticEvaluator which is designed to work regardless of the model type as long as you run the test_fn in your trainer step

  • The first two changes allow torch trainer steps to be followed by an agnostic evaluator step, see the example here.

  • Proposed a new naming scheme, which is now integrated into the built-in steps, in order to make it easier to handle feature/label names

  • Implemented a new adapted version of 2 TFX components, namely the Trainer and the Evaluator to allow the aforementioned changes to take place

  • Modified the TorchFeedForwardTrainer to showcase how to use TensorBoard in conjunction with PyTorch

Bug Fixes + Refactor

  • Refactored how ZenML treats relative imports for custom steps. Now:



Big shout out to @SarahKing92 in issue #34 for raising the above issues!


0.3.4


This release is a big design change and refactor. It involves a significant change in the Configuration file structure, meaning this is a breaking upgrade.
For those upgrading from an older version of ZenML, we ask to please delete their old pipelines dir and .zenml folders and start afresh with a zenml init.


If only working locally, this is as simple as:


cd zenml_enabled_repo
rm -rf pipelines/
rm -rf .zenml/



And then another ZenML init:


pip install --upgrade zenml
cd zenml_enabled_repo
zenml init



New Features


    

  • Introduced another higher-level pipeline: The NLPPipeline. This is a generic
NLP pipeline for a text-datasource based training task. Full example of how to use the NLPPipeline can be found here
    • 

  • Introduced a BaseTokenizerStep as a simple mechanism to define how to train and encode using any generic
tokenizer (again for NLP-based tasks).
    • 



Bug Fixes + Refactor


    

  • Significant change to imports: Now imports are way simpler and user-friendly. E.g. Instead of:
    • 



from zenml.core.pipelines.training_pipeline import TrainingPipeline


A user can simple do:


from zenml.pipelines import TrainingPipeline


The caveat is of course that this might involve a re-write of older ZenML code imports.


Note: Future releases are also expected to be breaking. Until announced, please expect that upgrading ZenML versions may cause older-ZenML
generated pipelines to behave unexpectedly.