More intelligent and flexible input and output tensor naming for TorchScript
Exporter
#62
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously, the
TorchScript
exporter directly enforced input and output tensor names follow theINPUT__{i}
OUTPUT__{i}
convention. The main reason for this is that triton expects certain naming conventions to deal with the fact that TorchScript models don't come with some necessary metadata triton needs to match input tensors to the appropriate ordering expected by the model. If your model only has one input and one output, this detail is irrelevant, and any naming will work.This convention is not the only way to deal with this (see here): one can also name the tensors by the parameter names in the models
forward
, or name the tensors with<name>__<index>
.In any case, the
INPUT__{i}
OUTPUT__{i}
tensor naming enforcement was leading to some difficulties when ensembling multiple torchscript models due to conflicting tensor names.This PR makes naming tensors in
TorchScript
models more flexible by doing the following:Input tensors
{tensor_name: shape}
for theinput_shapes
, the user passed names will be used. A warning will be given telling users to be careful when using their own name.List[shape]
toinput_shapes
, the names will be inferred directly from theforward
method of theTorchScript
model - this behavior is recommended by triton.Output tensors
List[name]
tooutput_names
, the user passed names will be usedoutput_names
is left asNone
, theOUTPUT__{i}
convention will be used