Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding A 'Batch' class #11

Closed
andthattoo opened this issue Oct 9, 2024 · 0 comments
Closed

Adding A 'Batch' class #11

andthattoo opened this issue Oct 9, 2024 · 0 comments
Assignees
@andthattoo
Labels
enhancement New feature or request

Comments

@andthattoo
Copy link
Member

We should have a 'BatchExecutor' class, that would execute a large set of inputs in parallel with the selected pipeline (or task).

e = BatchExecutor(pipeline=pipeline)
inputs = [TaskInput(), ...]  # of type TaskInput
await e.execute().export()

We should not collect Batch results in-memory data but we should have an export option to export generated data to a 'jsonl' or a specific database dump.
@andthattoo andthattoo self-assigned this Oct 9, 2024
@andthattoo andthattoo added the enhancement New feature or request label Oct 9, 2024
@andthattoo andthattoo changed the title Adding A 'distributor' class Adding A 'Batch' class Oct 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@andthattoo andthattoo

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@andthattoo @anilaltuner