Validate whether transferred model equals local model #265
Assignees
Labels
Comments
robin-oester
added a commit
that referenced
this issue
Oct 25, 2023
Currently, we use a naïve approach in order to store our trained models in the model storage. We want to have the possibility to do this more efficiently via pluggable policies. In particular, we want to support the following: - store first model of a pipeline with all layer information and other metadata - store subsequent models more efficiently by just storing weights/biases - implement other approach, using xor-difference between successive models and storing float exponent bits separately - create infrastructure to make these approaches pluggable and modularly expandable - make approaches configurable (with additional options) in pipeline config This should enable us to examine basic research questions in multi-model management. Corresponding issues: #293, #265
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When the trainer server uploads the model to the model storage, we could send a hash at the end to validate whether the file that the models storage received really equals the file that the trainer server wanted to send to identify if something went wrong.
We can use something like md5 hashing of big files as discussed here: https://stackoverflow.com/a/1131238/1625689
The text was updated successfully, but these errors were encountered: