Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update test cases to prevent platform-specific variability #487

Open
ztangent opened this issue Nov 9, 2022 · 0 comments
Open

Update test cases to prevent platform-specific variability #487

ztangent opened this issue Nov 9, 2022 · 0 comments

Comments

@ztangent
Copy link
Member

ztangent commented Nov 9, 2022

Per this comment, there are sometimes situations where Gen's test suite fails due to platform variability (e.g. 32-bit vs 64-bit systems) affecting random number generation even with fixed number generation, and hence approximate equality tests, such as the one below:

@testset "default proposal" begin
Random.seed!(1)
num_particles = 10000
ess_threshold = 10000 # make sure we exercise resampling
# initialize the particle filter
init_observations = choicemap((:x_init, obs_x[1]))
state = initialize_particle_filter(model, (1,), init_observations, num_particles)
# do steps
argdiffs = (UnknownChange(),) # the length may change
for T=2:length(obs_x)
maybe_resample!(state, ess_threshold=ess_threshold)
new_args = (T,)
observations = choicemap((:chain => (T-1) => :x, obs_x[T]))
log_incremental_weights, = particle_filter_step!(state, new_args, argdiffs, observations)
@test length(log_incremental_weights) == num_particles
end
# check log marginal likelihood estimate
expected_log_ml = log(hmm_forward_alg(prior, emission_dists, transition_dists, obs_x))
actual_log_ml_est = log_ml_estimate(state)
@test isapprox(expected_log_ml, actual_log_ml_est, atol=0.02)
end

This platform-specific variability happens despite fixing the random seed to 1. I'm not entirely sure what the most principled way to address this is, but the simple fix would be to relax the tolerance level (e.g. to 0.03 from 0.02) so that platform variability doesn't overly affect the result.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant