Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TensorRT EP] Enable more trt options #237

Merged
merged 32 commits into from
Mar 15, 2024
Merged

Conversation

yf711
Copy link
Collaborator

@yf711 yf711 commented Feb 8, 2024

This PR enables more ORT-TRTEP v2 options:

trt_max_partition_iterations
trt_min_subgraph_size
trt_dla_enable
trt_dla_core
trt_engine_cache_prefix
trt_dump_subgraphs
trt_force_sequential_engine_build
trt_context_memory_sharing_enable
trt_layer_norm_fp32_fallback
trt_timing_cache_enable
trt_force_timing_cache
trt_detailed_build_log
trt_build_heuristics_enable
trt_sparsity_enable
trt_builder_optimization_level
trt_auxiliary_streams
trt_tactic_sources
trt_extra_plugin_lib_paths
trt_profile_min_shapes
trt_profile_max_shapes
trt_profile_opt_shapes
trt_cuda_graph_enable
trt_cuda_graph_enable
trt_dump_ep_context_model
trt_ep_context_file_path
trt_ep_context_embed_mode

Check updated README for more detail

@yf711 yf711 requested a review from Tabrizian February 9, 2024 00:28
Copy link
Member

@Tabrizian Tabrizian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for contribution!

Is it possible to add some testing for it in this PR: https://github.com/triton-inference-server/server/blob/main/qa/L0_onnx_optimization/test.sh

README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
@yf711
Copy link
Collaborator Author

yf711 commented Feb 16, 2024

Thanks for contribution!

Is it possible to add some testing for it in this PR: https://github.com/triton-inference-server/server/blob/main/qa/L0_onnx_optimization/test.sh

Could you grant me access to server repo? I will init pull request after that

@dyastremsky
Copy link
Contributor

Thanks for contribution!
Is it possible to add some testing for it in this PR: https://github.com/triton-inference-server/server/blob/main/qa/L0_onnx_optimization/test.sh

Could you grant me access to server repo? I will init pull request after that

Thanks for being willing to add testing for this ticket! What sort of access do you need? There are pull requests from different users without permissions being granted. You should be able to fork the repository then submit the pull request.

@dzier
Copy link
Contributor

dzier commented Feb 27, 2024

@yf711 I have granted you access to submit your PR to the server repo.

@jywu-msft
Copy link

any updates? can we merge this PR first and then add tests in a separate PR?

@yf711
Copy link
Collaborator Author

yf711 commented Mar 13, 2024

Hi @dzier @Tabrizian Just a quick reminder that this PR has been finished. Could you test this PR on your CI?

Copy link
Contributor

@dyastremsky dyastremsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work! CI testing has passed.

dyastremsky

This comment was marked as duplicate.

@dyastremsky dyastremsky merged commit 72189f9 into main Mar 15, 2024
3 checks passed
@dyastremsky dyastremsky deleted the yifanl/moreTRTOptions branch March 15, 2024 16:12
@dyastremsky dyastremsky restored the yifanl/moreTRTOptions branch March 15, 2024 16:12
@yf711 yf711 deleted the yifanl/moreTRTOptions branch May 8, 2024 22:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants