Notes from 2016.06.22 meeting

Ideas for oed extensions

1. Confirm that OED doesn't work when differentiating between identical models (EIG is 0 for each experiment)
2. Better measure on uncertainty of EIG (variance doesn't work, too small)
3. Identifiability of A and B - A and B are unidentifiable if there's no way to distinguish between the two. A and B could perhaps make the same predictions on stimuli with different parameters (fiddling with parameters) but if the models have different priors on the parameters then we can begin to distinguish
4. Find a good experiment w/ continuous experiment space, and test out searching experiments with something other than Enumerate
5. Understanding more about EIG/AIG correlation, and what this means wrt your models
   1. Get benchmark for maximal correlation between EIG and AIG. There can't be a perfect correlation, since AIG will never equal EIG in every case.
   2. If you have a model close to the true model but maybe not precisely correct, can you tell from EIG/AIG correlation?
   3. EIG depends on how good your models are - predictions about IG will be off if you don't have the right model - the data is following a different process
6. Use OED for parameter estimation. Identify experiments that best help arrive at suitable parameter estimates. How would you do this?
   1. cf. mixture models
7. Staircasing (psychophysics)
   1. Compare staircasing to OED
   2. Using OED for prior elicitation
   3. Michael Franke Tubingen + Noah + MH - different methods of measuring priors (what does the crowd believe, CogSci 2016).
      1. Three methods: compare binary judgments, single judgment about continuous space, plausibility judgments about discretizations (histogramesque)
      2. Binary judgments aren't good? But sometimes necessary (e.g. kids)?
      3. Betting strategy doesn't work for, e.g. novel features, novel categories.
   4. "Sequential" OED