Initial Evaluation Mode Roadmap

[FelixTJDietrich]

Motivation

We want to evaluate the performance of modules for research purposes to better compare and iterate. This is a roadmap for the initial evaluation mode that at least @FelixTJDietrich wants to use for his Master's thesis.

Roadmap

• Evaluation Data Script #35
• Display Details in Module Requests #37
• Add module config for module_example and module_text_llm #44
• Improve playground requests and state management #52
• Improve feedback and some fixes #56
• More usable submission detail for evaluation mode #61
• Fix export script issues and add final evaluation data selection #64
• Fix download programming repositories #70
  • Settle on evaluation data to use for final evaluation
• Add evaluation mode running in batch mode #65
• Indicate grading instruction with badge #92
• Add experiment middleware and LLM call tags for logging #160
• Fix send training feedback getting stuck #166
• Update prompts and general improvements and fixes #170
• Add support for manual evaluation #171
• Add automatic evaluation with LLM-as-a-Judge, LangSmith export, and SGI evaluation #174