You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
For probes that don't compose their prompts, we can get info on which prompts are more or less successful in the bag of models used for calibration. This can be used to order prompts within a probe.
Once we have this infrastructure, we can then apply it to probes that compose their prompts out of components, as long as the component choice is logged in the attempt (not in this feature).
Process
update the calibration process to record (probe, detector, prompt, asr) tuples
from this derive (probe, prompt, asr) tuples taking the mean (not median) over all detectors used
work out a way of storing this artefact (HF? compressed and local? use a small hash like crc32? cut out everything below a certain ASR?)
work out where to add this function (probes.base.Probe() ?)
implement the sort
add a config variable for whether to do this optimisation
The text was updated successfully, but these errors were encountered:
Summary
For probes that don't compose their prompts, we can get info on which prompts are more or less successful in the bag of models used for calibration. This can be used to order prompts within a probe.
Once we have this infrastructure, we can then apply it to probes that compose their prompts out of components, as long as the component choice is logged in the attempt (not in this feature).
Process
probes.base.Probe()
?)The text was updated successfully, but these errors were encountered: