Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Review models architecture #55

Closed
7 tasks done
ben-jy opened this issue Aug 21, 2024 · 2 comments · Fixed by #75
Closed
7 tasks done

Review models architecture #55

ben-jy opened this issue Aug 21, 2024 · 2 comments · Fixed by #75
Assignees
Labels
enhancement New feature or request

Comments

@ben-jy
Copy link
Collaborator

ben-jy commented Aug 21, 2024

According to potential models that will be benchmarked, and the way we will benchmark them, architecture should be reviewed. We must define if it makes sense to create a new interface for benchmarked models, or if we use interface/class already defined in ontime (AbstractModel/Model, see https://github.com/ontime-re/ontime/tree/develop/src/ontime/core/modelling)

For the benchmark models, we need :

  • an evaluation method, to be performed in rolling window mode
  • a basic fit method, with training managed by the trained model
  • a predict method, which can predict on a new series as well as on the one the model has been trained on
  • a flag to determine the mode of evaluation (zero-shot, full-shot, few-shot)
  • the possibility to load a checkpoint, particularly for foundation models
  • univariate model should be able to perform multi-univariate predictions when given input series is multivariate
  • a means of resetting or reinstating the model between runs (Improve benchmark objects instanciation #68)

This issue in linked to issue #46

@ben-jy ben-jy changed the title evtl. review architecture (onTimeModel vs modelHolder, etc.) Review architecture (onTimeModel vs modelHolder, etc.) Aug 21, 2024
@ben-jy
Copy link
Collaborator Author

ben-jy commented Aug 21, 2024

For now, we created an interface AbstractBenchmarkModel (see https://github.com/ontime-re/ontime/tree/48-benchmarking-fix-zero-few-shot/src/ontime/module/benchmarking) that must implement every benchmark models. This class is completely independent from existing ontime model.
We must therefore define if we keep the code this way or if we reuse existing class.

@ben-jy ben-jy added the enhancement New feature or request label Aug 21, 2024
@ben-jy ben-jy added this to the v0.5.1 - Modelling and predictions milestone Aug 21, 2024
@ben-jy ben-jy changed the title Review architecture (onTimeModel vs modelHolder, etc.) Review architecture (onTimeModel vs BenchmarkModel, etc.) Nov 27, 2024
@ben-jy ben-jy changed the title Review architecture (onTimeModel vs BenchmarkModel, etc.) Review models architecture (onTimeModel vs BenchmarkModel, etc.) Nov 27, 2024
@ben-jy ben-jy changed the title Review models architecture (onTimeModel vs BenchmarkModel, etc.) Review models architecture Nov 29, 2024
@ben-jy ben-jy linked a pull request Dec 18, 2024 that will close this issue
@ben-jy
Copy link
Collaborator Author

ben-jy commented Dec 18, 2024

we do not need to specifically implement the possibility of loading of checkpoint, as we can let the specific wrapper to handle this with e.g. a "checkpoint" parameter.

@ben-jy ben-jy closed this as completed in #75 Jan 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants