Add test runner to CI

[nathan-weinberg]

@danmcp has introduced some tests in #9 - we should get testing setup as part of our current CI stack once they are merged in

[nathan-weinberg]

Note for this we will need a GPU-enabled runner with the following specs:

• 24GB vram for MT-Bench
• 96GB vram for MMLU

[russellb]

Note for this we will need a GPU-enabled runner with the following specs:

• 24GB vram for MT-Bench

This job should be capable:

instructlab/instructlab#1556

I can help get the same thing running on this repo, but it will take some additional repo configuration first.

• 96GB vram for MMLU

do you have an AWS instance type in mind? Ideally the minimal / cheapest type capable of testing this

[danmcp]

For the simple flow test, 24GB of vram should work for both. I haven't tried to make it work with less. We've been using g5.4xlarge.

[alimaredia]

I think we can make CI work with a g4dn.2xlarge instance. I'm going to give it a try today and report back on this issue. If not the g5.4xlarge should definitely work.

[booxter]

@alimaredia any updates on this issue? It would be nice to validate patches in the repo through CI before merging.

[alimaredia]

@booxter I don't have any updates. I've been focused on the instructlab CI lately. If someone wants to take the lead on establishing CI within the eval repo over the next week or so I'd be happy to review or tell that person what I see the next steps are.