Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for pipeline steps parallelization #312

Merged
merged 45 commits into from
Jan 29, 2024
Merged

Conversation

irux
Copy link
Contributor

@irux irux commented Aug 8, 2023

closes #177

@irux irux marked this pull request as draft August 8, 2023 14:57
@irux irux changed the base branch from main to v3 January 2, 2024 10:15
Copy link
Member

@disrupted disrupted left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please always label your PRs.

Also add a section about the new feature being in preview and not ready for production use in the docs.

@irux irux added the type/enhancement New feature or request label Jan 23, 2024
@irux
Copy link
Contributor Author

irux commented Jan 23, 2024

Also add a section about the new feature being in preview and not ready for production use in the docs.

Where do you think I should put it ?

@disrupted disrupted added component/generator Related to the generation of pipeline.yaml component/cli Related to the CLI commands component/handlers Related to component handlers labels Jan 23, 2024
@disrupted
Copy link
Member

Also add a section about the new feature being in preview and not ready for production use in the docs.

Where do you think I should put it ?

good point, I am not entirely sure. There's no dedicated section I can think of, but perhaps add it to the kpops --help? what do you think?

@irux
Copy link
Contributor Author

irux commented Jan 24, 2024

Also add a section about the new feature being in preview and not ready for production use in the docs.

Where do you think I should put it ?

good point, I am not entirely sure. There's no dedicated section I can think of, but perhaps add it to the kpops --help? what do you think?

but isn't already there?

@irux irux changed the title Add parallelization Add support for pipeline steps parallelization Jan 24, 2024
@disrupted
Copy link
Member

Also add a section about the new feature being in preview and not ready for production use in the docs.

Where do you think I should put it ?

good point, I am not entirely sure. There's no dedicated section I can think of, but perhaps add it to the kpops --help? what do you think?

but isn't already there?

where do you see that?
To me it only shows

  • --parallel / --no-parallel: Enable or disable parallel execution of pipeline steps. If enabled, multiple steps can be processed concurrently. If disabled, steps will be processed sequentially. [default: no-parallel]

Copy link
Member

@disrupted disrupted left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@irux irux merged commit 86f8db9 into main Jan 29, 2024
6 of 7 checks passed
@irux irux deleted the feature/parallelization branch January 29, 2024 07:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/cli Related to the CLI commands component/generator Related to the generation of pipeline.yaml component/handlers Related to component handlers type/enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Parallelize component steps execution
2 participants