-
-
Notifications
You must be signed in to change notification settings - Fork 63
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve Handling of cpp_options
#1022
base: master
Are you sure you want to change the base?
Conversation
It sounds like the actual issue here is that cmdstanr does not correctly identify whether model was compiled with stan_opencl, should we be changing that inside of the model class instead of changing this test? |
@SteveBronder did you see this draft PR that's related to this issue? #1023 Perhaps these should be combined? |
Yes I think it would be good to merge these two PRs together into one |
Sounds good, I'll polish that one up and combine them. Any idea if the opencl tests are actually running in CI (as in is Also trying to get hold of a windows machine so I can repo the windows unit test failures. If there's any way to run these on ubuntu with docker or similar, would love to hear. |
guessing the test failures are due to this: r-lib/actions#217 |
Sorry for the multiple pings in a day....but I found another related deepish problem. This test seems to be erroneously passing in the master branch: https://github.com/stan-dev/cmdstanr/blob/master/tests/testthat/test-threads.R#L162-L166 As in, both STAN_THREADS is not successfully getting set to MRE (from master branch):
The fix is that somewhere we need to convert falsy values at the R level to completely unset the ENV variable at the bash/cpp level. I'm not sure exactly where in the code it makes the most sense to do this, but I'll take a stab at it. Alternatively, we could inform users that they need to set |
No we shouldn't do anything here, since this is behaviour in Cmdstan - not cmdstanr. The options a user specifies should match those that are then set in |
ok, I'll update the test and the docs then. That's much easier. :-) |
Addresses several issues with the opencl tests: - Model object does not correctly identify whether model was compiled with stan_opencl. Therefore, models were not recompiled within the tests when recompilation is needed. Resolved this by forcing recompile. - Tests referenced a metadata attribute that was not defined. I've added checks for opencl related attributes that are set when opencl is functioning properly. - Version test used a data input that was never created.
* Model objects from exe file compiled with opencl fail to sample. * stan_threads set in makefile not respected
* mod$cpp_options() now shows options as false if unset (same as the `info` cli command). Previously, this was inconsistent: sometimes showing the user-provided arg and sometimes showing the output of `info` * tests now match expected behavior of cmdstan
Make it more clear that setting cpp_options = list(OPTION = FALSE) results in OPTION being set (turned on) rather than unset (turned off).
96cce2c
to
0eb126e
Compare
Sorry for the wall of text, but I need some input before moving forward.
It accomplishes:
Does not solve
Looking for input on two topics: 1. What should the behavior be when a model object is from an already compiled model that was compiled with different args than provided in the
|
cpp_options <- model_compile_info(self$exe_file()) |
One more wrinkle...prior to cmdstan v2.27.0, this "model info" does not exist. In that case, if the binary is already compiled, we have know way of knowing what flags were used. For that matter...I think we are assuming throughout that the version of cmdstan the user currently has configured is the same as the version the binaries were compiled with. Currently, from what I can tell, we are not checking this assumption at all. We could check using model info when available, but before 2.27, all bets are off. I'll have to think a bit about a clean way to resolve this. I'd love to hear if others have ideas. |
@andrjohns I'm a bit swamped with work at the moment and haven't had a chance to look into this yet. Do you by chance have time to comment on this? |
- Test new logic with mocked CLI
@jgabry @andrjohns I have a version I'm pretty happy with now. It stores what the user requested as cpp options and what the binary has configured separately, and (unless compile=FALSE) re compiles if there is a mismatch. I ran the CI on my fork and only the windows tests are failing. If someone on the team is able to give some feedback, especially on the change in functionality, that would be great! I'd love to get this in mergeable state. In any case, I will just use this version for my own purposes. |
Hi, @andrjohns, @SteveBronder, and @jgabry: @katrinabrock showed up at the Stan meeting this week. She's working on this PR. She says she can fix the Windows issue, but before doing that, needs feedback on whether her changes are targeting what you think the behavior should be. @katrinabrock: If you could remind everyone which specific things you need answered, that'd help. @mitzimorris is going to take a first pass at working on requirements with @katrinabrock, but then there will probably be some questions on the final R integration. |
@mitzimorris @SteveBronder Here's my explanation of this change as promised: Initial Symptom
My ChangesUser-facing Changes in
|
Thank you! These changes totally make sense now and handle a lot of the issues we have from dealing with a make build system. Monday I will give this a harder read through and have some Qs |
Apologies for the extended delay @katrinabrock! I'm catching back up on Stan dev this week, and have better access to an OpenCL system now. I'll put aside some time to review on Wednesday |
@katrinabrock hi my apologies for the delay. Looking at the tests it looks like the new ones for the compile options are failing on windows? |
Hi @SteveBronder, Thanks for taking a look. That's correct. As mentioned in @bob-carpenter's comment above, right now, I'm not looking for approval of the code. I'd like to make sure that the functionality I'm trying to implement is the functionality desired by the core dev team. Does everything I've listed in the comment above from three weeks ago seem like the right set of changes? If so, I'll go ahead and diagnosis and resolve the windows issue. If you would like me to make a different set of logic changes, I'd like to settle on what the logic should be and implement that before worrying about the cross platform aspect. |
Thanks, @katrinabrock, that sounds totally reasonable. |
@SteveBronder when you have a chance do you mind taking another look, based on @katrinabrock's most recent comment. I think you're much more familiar with all these CmdStan C++ options than I am. It sounded like @katrinabrock was looking for confirmation that the outlined approach in #1022 (comment) is reasonable. It seems reasonable to me but it would be good to get either you (Steve) or @andrjohns to confirm too. |
I think @mitzimorris could also look over the logic and see if there are any concerns since we would likely want to have cmdstanpy and cmdstanr in sync. |
I think the logic is sound, w/r/t a compiled model. Re CmdStanPy, OpenCL support hasn't been implemented - see discussion here: https://discourse.mc-stan.org/t/cmdstanpy-with-gpu-opencl/28064/1 CmdStanPy's |
Thanks @mitzimorris for checking the logic. And thank you @katrinabrock for working on this!! It's great to have new contributors. |
@mitzimorris Thanks for the input! @jgabry Between you and mitzi, do you think this is enough approval to go forward with this logic? Or do we still need @SteveBronder or @andrjohns to look at it? |
I trust @mitzimorris' judgement on this, so I think it's ok to proceed. It would be great if @SteveBronder can help review the PR when it's ready since he knows a lot more about these particular aspects of CmdStan than I do. @andrjohns would also be great if he has the time (I know he's been busy lately). Thanks again @katrinabrock for working on this! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Apologies for my slow review! I have a few design questions below. But overall this is a very nice fix for the issue. I'll continue with a bit of review tomorrow. I still need to read over a lot of the tests
@@ -230,10 +231,13 @@ CmdStanModel <- R6::R6Class( | |||
stanc_options_ = list(), | |||
include_paths_ = NULL, | |||
using_user_header_ = FALSE, | |||
precompile_cpp_options_ = NULL, | |||
precompile_cpp_options_ = list(), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a reason these must be list()
and not null like the other argument defaults? Since you also use %||% later with a list()
being the rhs
(same q for all the other list()
args)
} | ||
self$exe_file(cmdstan_ext(strip_ext(exe_base))) | ||
if (dir.exists(self$exe_file())) { | ||
stop("There is a subfolder matching the model name in the same folder as the model! Please remove or rename the subfolder and try again.", call. = FALSE) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this cause a failure later when we try calling the executable?
} | ||
} | ||
|
||
# exe_info is updated inside the compile method (if compile command is run) | ||
exe_info <- self$exe_info(update = TRUE) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you rename this variable? It gets a little confusing having objects and functions with the same name
@@ -328,9 +353,72 @@ CmdStanModel <- R6::R6Class( | |||
} | |||
private$exe_file_ | |||
}, | |||
exe_info = function(update = FALSE) { | |||
if (update) { | |||
if (!file.exists(private$exe_file_)) return(NULL) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should this throw a warning (or even an error) if we ever attempt to get exe info from something that does not exist yet?
|
||
# Check recompilation upon changing header | ||
file.create(file_that_exists) | ||
with_mocked_cli(compile_ret = list(status = 0), info_ret = list(), code = expect_no_mock_compile({ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could we keep the line numbers to less than 85 or so? Runs over my screen here. @jgabry does cmdstanr have a linter or styler setup?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We haven't been running any linters on the code, although we could start doing that. Either way, I agree about keeping the lines short. There are likely some other places in the code where that needs to be fixed, although I've tried to keep them under 80 in the code I've written.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah, since it doesn't seem like a linter was run in the past, I didn't run one on my code either, but that makes it surely inconsistent. I think there is a way to run linter just on the diff so at least we don't make the problem worse. I can look into it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think there is a way to run linter just on the diff so at least we don't make the problem worse. I can look into it.
If that's simple to do then that sounds good, otherwise don't worry about it. We can (and probably should) start running a linter on the whole package, maybe after merging this PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TBH - I wouldn't advise that. It basically creates a huge diff so your subsequent blames become less meaningful. I think there is a way to exclude certain commits from blame...but....
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm yeah that’s a good point
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We did this years ago in the stan math library. We essentially have a huge file that tells the compiler to ignore any auto changes.
For cmdstanr we could set up a hook so that the linter runs before any future commits. So we would put the one global change in the .git-blame-ignore-revs
then have a git hook that runs precommit to run the linter
https://github.com/stan-dev/math/blob/develop/.git-blame-ignore-revs
But again, seperate issue from the PR
@katrinabrock if you can try to be mindful of not having too long of lines that would be great, but no need to go through counting column
# Check recompilation upon changing header | ||
file.create(file_that_exists) | ||
with_mocked_cli(compile_ret = list(status = 0), info_ret = list(), code = expect_no_mock_compile({ | ||
mod$compile(quiet = TRUE, user_header = tmpfile) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry I'm just not familiar with testthat
's mocking schema. Is mod
here the mod
from line 36?
mod <- cmdstan_model(
stan_file = testing_stan_file("bernoulli_external"),
exe_file = file_that_exists,
user_header = tmpfile
)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
um....good question. I don't think so because that shouldn't be in scope anymore. Which begs the question "where is this mod
object coming from. I'll look into it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lol thank you. Odd!!
mod_w_include <- cmdstan_model(stan_file = stan_program_w_include, compile=TRUE, | ||
include_paths = test_path("resources", "stan")) | ||
include_paths = test_path("resources", "stan"), | ||
force_recompile = TRUE) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is the side effect at issue here? I am just reading the code right now but tomorrow should have time to run the tests
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
😬 That's a mystery I didn't solve completely. Some test above was causing a side effect. (I think I have a version of the code somewhere that marked which test that was.) I realized actually there could be many such cases that are not caused by my change, but revealed by my change. In this situation (and some others), I "fixed" the problem by added force_recompile = TRUE
to the impacted test instead of actually cleaning up the culprit test.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, based on looking at my git history for when I was attempting to debug this. This is the test causing this particular side effect:
https://github.com/stan-dev/cmdstanr/blob/master/tests/testthat/test-model-compile.R#L85
So if you uncomment force_recompile, there will be a failure. If you then comment out the above linked test, the later test will pass without recompilation. (If I recall correctly.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh my. Okay it sounds like something is carrying state through the tests. I had another project yesterday to work on. Friday I can give this a whirl
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the thing that's carrying state through tests is the binary....because recompiling fixes it.
Thanks @SteveBronder, much appreciated! And thanks in advance for continuing tomorrow. I responded to a few of the comments that were either directed at me or that I knew the answer to, I'll let @katrinabrock address the others.
Great, this is definitely something I trust your judgment on more than mine! |
@SteveBronder Thanks for the review. I responded to the comments where I had something to add from memory. Most of them either I will update the code based on your suggestion or I need to look a little closer and rerun the code to properly answer you question. It will probably be a few weeks before I can devote a chunk of time to updating everything. With the mocks, I think it will make sense if you play around with them. They are doing what I want, but certainly can be designed better. |
Sounds good. Thanks again for working on this @katrinabrock! |
All good and thank you for the PR! I should have time Friday to mess with the test issues |
Addresses several issues with the opencl tests:
Submission Checklist
Summary
Please describe the purpose of the pull request.
Copyright and Licensing
Please list the copyright holder for the work you are submitting
(this will be you or your assignee, such as a university or company):
Max Planck Institute of Animal Behavior
By submitting this pull request, the copyright holder is agreeing to
license the submitted work under the following licenses: