Remove attributes from PipelineContext that can change per Block in a Pipeline #491
Labels
enhancement
New feature or request
epic
Larger tracking issue encompassing multiple smaller issues
UX
Affects the User Experience
The following attributes of
PipelineContext
are specific to individualBlocks
in aPipeline
, and should not be set in the overallPipelineContext
:model_id
- Not allBlocks
use a model at all, and those that do may choose to use separate models for separateBlocks
within a singlePipeline
. We have some upcoming use-cases that expect to use different models within a singlePipeline
, so we need to get in front of that.model_family
- same reasoning as the above - onePipeline
could have separateBlocks
that call into mixtral, granite, or other models within a singlePipeline
num_instructions_to_generate
- This is only used to substitute then
parameter ingen_kwargs
of anLLMBlock
's config when it has the special value ofscaled
. However, users may want a different value perBlock
here and it's better to show them how to edit this in theirPipeline
yaml as opposed to having a single parameter that they may not want to apply to allBlocks
.max_num_tokens
- This is only used to substitute themax_tokens
parameter ingen_kwargs
of anLLMBlock
's config when it has one of a few predefined block names. That's brittle, and users will need the ability to adjust the tokens for anyLLMBlock
independently in aPipeline
without having to rely on special predefined block names to control the maximum tokens generated.All of these must be specified per-block instead, which today means in the yaml configuration of the
Block
.What this will leave on
PipelineContext
is:client
- the OpenAI Client - optional, as not all Pipelines need to call an inference endpointdataset_num_procs
- the number of parallel threads to use when processing datasets - optional, with a reasonable defaultbatch_num_workers
- the number of parallel threads to use when calling inference servers - optional, with a reasonable defaultbatch_size
- the batch size to use in completion requests when calling inference endpoints that support batching - optional, with a reasonable defaultcheckpoint_dir
- the directory to store checkpoints during data generation, enabling some ability to recover from where we left off if a data generation gets interrupted - optional, with checkpointing disabled if not specifiedThe text was updated successfully, but these errors were encountered: