feat: support third-party oneflow device extension #549
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
also, refactor the build process of model and tokenizer using pretrained_model_path cofnig
0x404
force-pushed
the
zqh/llama_device
branch
from
49a5ff2
to
0ece33a
Compare
ShawnXuan
approved these changes
Sep 4, 2024
ShawnXuan
added a commit
that referenced
this pull request
Sep 5, 2024
* update llama for multi devices * xpu and npu config files * update device for inference * update * update * update README * update * format * format * fix * feat: support third-party oneflow device extension (#549) * feat: support third-party device oneflow extentions also, refactor the build process of model and tokenizer using pretrained_model_path cofnig * refactor: remove unnecessary config and warnings * docs: update readme for commands to run llama on npu and xpu * fix import order * update * update * fix: skip lint on oneflow third-party imports --------- Co-authored-by: Qunhong Zeng <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes of this PR made:
_init_distributed_env, import oneflow’s third-party library (i.e., oneflow-npu and oneflow-xpu) based on the actual device in use, because _DistributeUtil is only called once, thus we will import the required third-party library only once.BasePipeline: if the user provides themodel_pathparameter, thenBasePipelinewill load the model based on this parameter (which has a higher priority than the model configuration in config’smodel.cfg.pretrained_model_path). If this parameter is not provided, the default will be to usemodel.cfg.pretrained_model_path.tokenization.tokenizerin the config does not have apretrained_model_pathset, the default will be to use the filetokenizer.modelundermodel.cfg.pretrained_model_path(this is usually correct, as it is the default storage location and naming used by hugging face).