added simple model compare example #88
Conversation
Benchmark Results
Benchmark PlotsA plot of the benchmark results have been uploaded as an artifact to the workflow run for this PR. |
|
Kind of irrelevant to that PR, but just thought I'd mention that we recently implemented an automated reporting for loo comparisons in R: library(brms)
library(report)
m1 <- brms::brm(mpg ~ qsec, data = mtcars, refresh=0)
m2 <- brms::brm(mpg ~ qsec + drat, data = mtcars, refresh=0)
m3 <- brms::brm(mpg ~ qsec + drat + wt, data = mtcars, refresh=0)
x <- brms::loo_compare(
brms::add_criterion(m1, "loo"),
brms::add_criterion(m2, "loo"),
brms::add_criterion(m3, "loo"),
model_names = c("m1", "m2", "m3")
)
report(x)
#> The difference in predictive accuracy, as indexed by Expected Log Predictive
#> Density (ELPD-LOO), suggests that 'm3' is the best model (ELPD = -79.19),
#> followed by 'm2' (diff-ELPD = -13.30 +- 4.42, p = 0.003) and 'm1' (diff-ELPD =
#> -23.01 +- 4.43, p < .001)report(x, include_IC = FALSE)
#> The difference in predictive accuracy, as indexed by Expected Log Predictive
#> Density (ELPD-LOO), suggests that 'm3' is the best model (ELPD = -79.19),
#> followed by 'm2' (diff-ELPD = -13.30 +- 4.42, p = 0.003) and 'm1' (diff-ELPD =
#> -23.01 +- 4.43, p < .001)report(x, include_ENP = TRUE)
#> The difference in predictive accuracy, as indexed by Expected Log Predictive
#> Density (ELPD-LOO), suggests that 'm3' is the best model (LOOIC = 158.39, ENP =
#> 5.14), followed by 'm2' (diff-ELPD = -13.30 +- 4.42, p = 0.003, LOOIC = 184.98,
#> ENP = 3.97) and 'm1' (diff-ELPD = -23.01 +- 4.43, p < .001, LOOIC = 204.42, ENP
#> = 2.52)Created on 2024-07-08 with reprex v2.0.2 The interesting thing we add is a p-value, based on the z-approximation of ELPD diff and ELPD diff SE (but I'm not sure the SE is returned by PSILOO for Julia). |
|
Oh, that is quite neat! Thanks for sharing. I believe that they are. Also do you have any papers/preprint on adding that p-value? |
|
No paper but see the details section (https://easystats.github.io/report/reference/report.compare.loo.html#details) basically it comes from a Stan thread where Vehtari was showing how the ELPD diff is normally distributed, and how the ELPD SE can be used to get a standardized z-ELPD-diff. Once you have a z-like value, getting the p-value becomes trivial
And I dream of seeing a whole easystats.jl :p one day maybe |
|
@kiante-fernandez, aweome! This looks like it will be a useful addition to the documentation. Thanks for the PR. I will review by the end of the week. @DominiqueMakowski, that looks very useful. I could see a use for something similar in Julia, perhaps Report.jl, which would work as a general inferface across the statistical packages. |
|
@kiante-fernandez, thanks for the PR. Please make sure to run the documentation locally. There were several errors (despite the tests showing the docs were built correctly). One was that Paretosmooth was not added to the Project.toml in the docs folder. There were a few other minor errors. I also removed the for loop in the turing model to make the example simpler. For some reason, my changes did not merge into this PR. So I will add those through a seperate PR. |
just added a working example of comparing models using ParetoSmooth.jl