Validate ABC-SMC parameter estimation on BiSSE model

[MathiasRenaud]

No description provided.

[MathiasRenaud]

All the kinks should be worked out of BiSSE model now. It run with 10,100, and 1000 particles. With all 6 priors set as a gamma distribution(rate=2, shape=1) and all proposals set to normal distribution (mean=0, sd=0.1), all parameters were overestimated (priors need tuning), but it runs:

[ArtPoon]

Work on this issue will also be dependent on resolving parallelization #26

[ArtPoon]

@helenhe96 to try re-running with different prior specifications. For a sanity check, try priors that are tightly centered around the actual values.

[helenhe96]

The simulation took 4624 seconds (77 minutes) on one thread.

[helenhe96]

If using priors centered around the true values (with normal distribution), the posteriors are good representations of the true values:

[ArtPoon]

Okay! But I'm guessing that we've "locked in" the priors tightly around the true parameter values. The next step is to relax the variance in the priors, and deflect the initial parameter values away from the true values.

[ArtPoon]

When you run this analysis, can you please use multiple threads?

[helenhe96]

When I use multiple threads on this my computer crashes... I have tried 10 threads and 5 threads.

[gtng92]

Issue #131

[ArtPoon]

From dev meeting:

• relaxing prior distribution for one parameter of BiSSE model is tolerated (if possible, break down by parameter)
• relaxing more than one prior distribution is not tolerated well

Next steps:

• look for correlations between parameters in final distribution of particles, which would suggest that the parameters are confounded (cannot be estimated separately)
• check if you are rescaling branch lengths (setting normalize to mean or median). If so, try setting to none. (Apparently it is set to none)
• run through experiments systematically and summarize in a short report with knitr. Use the following sections:
  • objectives
  • experiment 1 -- priors narrow and centered at true values
  • experiment 2 -- priors broad and centered at true values
    • look for correlations among particles wrt. pairs of parameters
    • evaluate effect of tree size (data)
  • experiment 3 -- priors narrow and centered at true values except one (broad and shifted)
    • cycle through each parameter
  • experiment 4 -- relax two priors at a time, focus on most identifiable

[ArtPoon]

Pending implementation of labelled tree kernels, see #133

[ArtPoon]

Before working through the proposed experiments above, better do some checks on responsiveness of kernel distances (and perhaps other tree distances) to varying BiSSE model parameters.

1. Simulate trees where one parameter is varied and the other model parameters are fixed (will need to determine what are "informative" parameter settings here).
2. Select one tree to be the "target" tree and calculate distances of other simulations to the target.
3. Plot distances versus parameter value, hopefully we see a concave trend where distance reaches a minimum near the true parameter value.