Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

.extract_model_data(): Warn that arguments wrhs and orhs are ignored #1575

Merged
merged 1 commit into from
Dec 18, 2023

Conversation

fweber144
Copy link
Contributor

This PR adds warnings that arguments wrhs and orhs are currently ignored by .extract_model_data() (the workhorse function for extract_model_data() defined in get_refmodel.brmsfit()). This is relevant for projpred::proj_linpred(), projpred::proj_predict(), and projpred:::predict.refmodel(): When using an rstanarm reference model, it would be possible to specify weights and offsets in these 3 functions via arguments weightsnew and offsetnew, respectively. When using a brms reference model, these arguments are ignored (weights and offsets are fetched from newdata, as brms does at other places, too). So far, this has only been documented at ?get_refmodel.brmsfit. With this PR, warnings are thrown to increase awareness.

Illustration:

data("df_gaussian", package = "projpred")

df_gaussian_new <- df_gaussian[30:33, ]
dat_new <- data.frame(y = df_gaussian_new$y, df_gaussian_new$x)
dat_new$wcol <- rep_len(6:5, nrow(dat_new))
dat_new$ocol <- dat_new$X6

df_gaussian <- df_gaussian[1:29, ]
dat <- data.frame(y = df_gaussian$y, df_gaussian$x)
dat$wcol <- rep_len(4:1, nrow(dat))
dat$ocol <- dat$X6

devtools::load_all() # use `library(brms)` if PR version is installed
bfit <- brm(
  y | weights(wcol) ~ X1 + X2 + X3 + X4 + X5 + offset(ocol), data = dat,
  chains = 1, iter = 500, seed = 7403, refresh = 0
)

library(projpred)
prj <- project(bfit, predictor_terms = paste0("X", 1:3), ndraws = 25,
               verbose = FALSE, seed = 4678)

# Intended usage (no warnings):
pln <- proj_linpred(prj, newdata = dat_new)
# Unintended usage (for a brms reference model, not for an rstanarm reference
# model); throws warnings:
pln_vec <- proj_linpred(prj, newdata = dat_new,
                        weightsnew = dat_new$wcol,
                        offsetnew = dat_new$ocol)
pln_fml <- proj_linpred(prj, newdata = dat_new,
                        weightsnew = ~ wcol,
                        offsetnew = ~ ocol)

When running the tests with this PR, I received the following error, which doesn't seem to be related to projpred and this PR, though:

* checking tests ... [248s/248s] ERROR
  Running ‘testthat.R’ [248s/248s]
Running the tests in ‘tests/testthat.R’ failed.
Last 13 lines of output:
  
  ══ Failed tests ════════════════════════════════════════════════════════════════
  ── Error ('tests.make_stancode.R:2471:3'): threaded Stan code is correct ───────
  Error: Syntax error found! See the message above for more information.
  Backtrace:
      ▆
   1. └─brms::make_stancode(bform, dat, family = student(), threads = threads) at tests.make_stancode.R:2471:3
   2.   └─brms:::.make_stancode(...)
   3.     └─brms:::parse_model(scode, backend, silent = silent)
   4.       └─brms (local) .parse_model(model, ...)
   5.         └─out$check_syntax(quiet = TRUE)
  
  [ FAIL 1 | WARN 0 | SKIP 6 | PASS 3017 ]
  Error: Test failures
  Execution halted

@fweber144
Copy link
Contributor Author

The conditioning on projpred version >= 2.8.0 in line https://github.com/fweber144/brms/blob/725590767c05711f4aa36412f348dc53eb53a3c8/R/projpred.R#L249 is done because in version 2.8.0 (more precisely, stan-dev/projpred#486), some adjustments were performed that allow brms to throw the warnings correctly.

@paul-buerkner
Copy link
Owner

Thanks!

@paul-buerkner paul-buerkner merged commit e496ca1 into paul-buerkner:master Dec 18, 2023
0 of 5 checks passed
@fweber144 fweber144 deleted the wrhs_orhs branch December 18, 2023 07:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants