Final evaluation notebook

[menamerai]

• Specify the model names (e.g. llama2_100k, llama2_200k,... or list with different checkpoints)
• Load next logprobs data (for each of the model names)
  → for now we just load a slice of the dataset (to develop faster)
• Specify the tokens we are interested (a sort of dynamic category)
  → using Rai’s GUI (need to be tested when integrate, just try to use the selected_ids as a dynamic variable and see how it turns out for now)
• Compute the statistics from the plot (copy form end2end.ipynb , currently commented out)
  → these calculate the percentiles for a group of tokens (which should be arbitrary)
• Plot: using function in delphi.eval.vis_per_token_model
• Specify 2 models for comparison.
• Get prompt examples (Rai’s notebook model_diff.ipynb)

[menamerai]

The notebook will be called eval_notebook.ipynb for now. Working on the first 2 bullet points today. I'll try to test this on Colab as well.

Post-note: did the first two bullets, also got started on third bullet. Unsure if it will work with the current ecosystem, to be tested when integrate with visualization functions.

[jettjaniak]

we have a PR and a trello card, we don't need this issue