Model wrapper with local fine-tuning

[zyzhang1130]

---

name: Pull Request
about: Create a pull request

Description

Added features to download models from hugging face model hub/load local hugging face model and fine-tune loaded model with hugging face dataset. Model loading and fine-tuning can happen both at the initialization stage and after the agent has been initialized (see README in agentscope/examples/conversation_with_agent_with_finetuned_model for details). Major changes to the repo include creating the example script conversation_with_agent_with_finetuned_model.py, adding a new model wrapper HuggingFaceWrapper in agentscope/examples/conversation_with_agent_with_finetuned_model/huggingface_model.py, and creating a new agent type Finetune_DialogAgent in 'agentscope/examples/conversation_with_agent_with_finetuned_model/finetune_dialogagent.py'. All changes are done in a new example directory agentscope/examples/conversation_with_agent_with_finetuned_model.

To test, run agentscope/examples/conversation_with_agent_with_finetuned_model/conversation_with_agent_with_finetuned_model.py by following the instructions in the README in the same directory.

Checklist

Please check the following items before code is ready to be reviewed.

• Code has passed all tests
• Docstrings have been added/updated in Google Style
• Documentation has been updated
• Code is ready for review

[ZiTao-Li]

See the inline comments. Besides, you can run pre-commit run --all-files to check if your coding style satisfies our repo's requirement.

[ZiTao-Li]

To be consistent with the other model wrappers, the 'raw' should contain the generated_text as well. Also, the

[zyzhang1130]

Updated. What was the second point?

[zyzhang1130]

Need to disable E203 for pre-commit run. It automatically reformats `examples/load_finetune_huggingface_model/huggingface_model.py' line147 by adding whitespace before ':' and then flags it as an error.

[ZiTao-Li]

• About the flask8 code style error on github, maybe the link can can help https://flake8.pycqa.org/en/3.1.1/user/ignoring-errors.html#in-line-ignoring-errors

• try to split the the agent to an separated file to make it more clean.

• change the name of this folder "load_finetune_huggingface_model" to "conversation_with_agent_with_finetuned_model" to make the naming style consistent.

[ZiTao-Li]

As a better practice, please create new branches (in your forked repo) for the updates on reformatting examples tasks (one branch for one example). Please do not let the changes affects each other.

[ZiTao-Li]

The Description of this PR seems outdated, please update it accordingly.

[zyzhang1130]

The Description of this PR seems outdated, please update it accordingly.

Done.

[DavdGao]

Please see inline comments, and:

1. It's not clear who is responsible to fine-tune the model, the agent or model wrapper? Since the new model configuration has all required arguments for fine-tuning, why don't we just implement the fine-tuning functionality all in the new model wrapper class (e.g. the constructor function)? So that all other agents in AgentScope can re-use the model wrapper and fine-tune their local models without modifications.

[DavdGao]

Since the model is loaded by device_map as auto, will there be any conflict between the attribute self.device (and device argument in the constructor)?

[zyzhang1130]

I presumed fine-tuning of LLMs happens on gpus, since it is (as far as I'm aware) infeasible to do on cpu. self.device is meant for device for inference only.

[zyzhang1130]

This part is updated. If the user didn't specify device in model_configs, device_map="auto" by default; otherwise device_map is set to the user-specified device. (see 8685213)

[DavdGao]

How about using the argument model_name_or_path like what transformers library does in from_pretrained function rather than providing two arguments?

[zyzhang1130]

Changed according to suggestion.

[DavdGao]

suggested to remove lines 56-59

[zyzhang1130]

I wanted to give the user the flexibility to specify the arguments needed in a json file, but also wanted to provide some explanation which cannot be done with json file, so I have two identical model_configs, one in agentscope/examples/conversation_with_agent_with_finetuned_model/conversation_with_agent_with_finetuned_model.py and the other in agentscope/examples/conversation_with_agent_with_finetuned_model/configs/model_configs.json.

[DavdGao]

The design is a little strange, and the interface should be more concise.

1. We have already specified the model configuration in agentscope.init and providemodel_config_name in line 66, why we have to repeat to set model_id in line 70 and 74?
2. Similar issue with dataset name in line 40 and 86.

[zyzhang1130]

This is an optional step where the user can choose to load another model after the agent has been instantiated if needed, depending on their use cases. Added a comment to clarify its purpose.

[DavdGao]

Maybe we should support user customized saving directory.

[zyzhang1130]

Added a new argument 'output_dir for the users to customize saving directory. By default will save to the save example directory if left unspecified.

[zyzhang1130]

Please see inline comments, and:

1. It's not clear who is responsible to fine-tune the model, the agent or model wrapper? Since the new model configuration has all required arguments for fine-tuning, why don't we just implement the fine-tuning functionality all in the new model wrapper class (e.g. the constructor function)? So that all other agent in AgentScope can re-use the model wrapper and fine-tune their local models without modifications.

Reloading and fine-tuning models after an agent has been instantiated need to introduce a new method to the agent class. If this is not required, this model wrapper in principle supports loading and fine-tuning for other types of agents.

[DavdGao]

Please see inline comments, and:

1. It's not clear who is responsible to fine-tune the model, the agent or model wrapper? Since the new model configuration has all required arguments for fine-tuning, why don't we just implement the fine-tuning functionality all in the new model wrapper class (e.g. the constructor function)? So that all other agent in AgentScope can re-use the model wrapper and fine-tune their local models without modifications.

Reloading and fine-tuning models after an agent has been instantiated need to introduce a new method to the agent class. If this is not required, this model wrapper in principle supports loading and fine-tuning for other types of agents.

Yes, and my question is that why we need to reloading and fine-tuning models within the agent? In my view, once we create a huggingface model wrapper, we have already decided to fine-tune the model. So that the training can be finished automatically within the constructor function of the model wrapper as follows?

class HuggingFaceWrapper(ModelWrapperBase):
     def __init__(self, *args, **kwargs): 
         # load model
         self.model = self.load_model(xxx)

         # fine tuning
         self.fine_tune_model()

    # ...

In this way, the agent doesn't need to do anything, and developers only need to set their model configuration.

[zyzhang1130]

Please see inline comments, and:

1. It's not clear who is responsible to fine-tune the model, the agent or model wrapper? Since the new model configuration has all required arguments for fine-tuning, why don't we just implement the fine-tuning functionality all in the new model wrapper class (e.g. the constructor function)? So that all other agent in AgentScope can re-use the model wrapper and fine-tune their local models without modifications.

Reloading and fine-tuning models after an agent has been instantiated need to introduce a new method to the agent class. If this is not required, this model wrapper in principle supports loading and fine-tuning for other types of agents.

Yes, and my question is that why we need to reloading and fine-tuning models within the agent? In my view, once we create a huggingface model wrapper, we have already decided to fine-tune the model. So that the training can be finished automatically within the constructor function of the model wrapper as follows?

class HuggingFaceWrapper(ModelWrapperBase):
     def __init__(self, *args, **kwargs): 
         # load model
         self.model = self.load_model(xxx)

         # fine tuning
         self.fine_tune_model()

    # ...

In this way, the agent doesn't need to do anything, and developers only need to set their model configuration.

I have a long-term use case scenario in mind, where there might be new data becoming available after the agent has been deployed (i.e., continual learning setting). In this case, the agent might need to be fine-tuned (potentially multiple times) after deployment. One such example we touched on during the discussion is fine-tuning the agents from their interaction traces in a multi-agent setup. Perhaps I can rename the agent class to better reflect this use case?

[DavdGao]

Please see inline comments, and:

1. It's not clear who is responsible to fine-tune the model, the agent or model wrapper? Since the new model configuration has all required arguments for fine-tuning, why don't we just implement the fine-tuning functionality all in the new model wrapper class (e.g. the constructor function)? So that all other agent in AgentScope can re-use the model wrapper and fine-tune their local models without modifications.

Reloading and fine-tuning models after an agent has been instantiated need to introduce a new method to the agent class. If this is not required, this model wrapper in principle supports loading and fine-tuning for other types of agents.

Yes, and my question is that why we need to reloading and fine-tuning models within the agent? In my view, once we create a huggingface model wrapper, we have already decided to fine-tune the model. So that the training can be finished automatically within the constructor function of the model wrapper as follows?

class HuggingFaceWrapper(ModelWrapperBase):
     def __init__(self, *args, **kwargs): 
         # load model
         self.model = self.load_model(xxx)

         # fine tuning
         self.fine_tune_model()

    # ...

In this way, the agent doesn't need to do anything, and developers only need to set their model configuration.

I have a long-term use case scenario in mind, where there might be new data becoming available after the agent has been deployed (e.g., continual learning setting). In this case, the agent might need to be fine-tuned (potentially multiple times) after deployment. Perhaps I can rename the agent class to better reflect this use case?

Okay, I understand your consideration. For me, agent.model.fine_tune() in continual learning setting is also acceptable. Whatever, please ensure the other agents can directly use this huggingface model configuration without modifying their code. Others plz see inline comments.

[zyzhang1130]

Further update of this pull request can be found here as it was moved to a new branch #240