Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove Turing integration tests #733

Open
wants to merge 15 commits into
base: master
Choose a base branch
from
Open

Conversation

penelopeysm
Copy link
Member

@penelopeysm penelopeysm commented Nov 29, 2024

Closes #703

General breaking changes

Breaking changes in TestUtils.DEMO_MODELS

  • What was originally demo_assume_observe_literal has been renamed to demo_assume_multivariate_observe_literal to avoid name clash with new model
  • demo_assume_literal_dot_observe renamed to demo_assume_dot_observe_literal to match the naming pattern of the other models
  • Added a new demo_assume_observe_literal, which is a univariate assume
@model function demo_assume_observe_literal()
    # univariate `assume` and literal `observe`
    s ~ InverseGamma(2, 3)
    m ~ Normal(0, sqrt(s))
    1.5 ~ Normal(m, sqrt(s))
    2.0 ~ Normal(m, sqrt(s))

    return (; s=s, m=m, x=[1.5, 2.0], logp=getlogp(__varinfo__))
end

This is a slightly annoying change, because anybody who uses demo_assume_observe_literal will have the behaviour changed silently. But the only code that uses it is DynamicPPL and Turing, so I think it's fine, as long as I fix it upstream too. (See GitHub-wide search for demo_assume_observe_literal)

Changes to test-internal utils

  • Added a function make_chain_from_prior([rng,] model, n_iters) which constructs an MCMCChains.Chains object by sampling from the prior of modelforn_iters` iterations.

Note this is in test/test_util.jl rather than src/test_utils/foo.jl, hence only accessible inside the test suite.

Changes to test suite

  • Removed the old Turing integration tests (see Turing integration tests should be moved to (or included in) Turing repo #703).
  • Most of them have been incorporated back into the DynamicPPL test suite itself. The functionality that required Turing.jl was either sampling (which has been replaced with the make_chain_from_prior function above), or some sampling algorithm with some space parameter (which has been replaced with a dummy algorithm MyAlg in test/varinfo.jl).
  • The remaining tests that haven't been added back are being tracked here: Update for proposed DynamicPPL 0.32 Turing.jl#2419 However, I think this can be part of a different PR as it would require more extensive modifications to our DemoModels module.

Feedback needed:

  • make_chain_from_prior seems too useful to languish in test/test_util.jl. Should this be in DynamicPPLMCMCChainsExt?

@coveralls
Copy link

coveralls commented Nov 29, 2024

Pull Request Test Coverage Report for Build 12302036826

Details

  • 22 of 22 (100.0%) changed or added relevant lines in 1 file are covered.
  • 22 unchanged lines in 5 files lost coverage.
  • Overall coverage decreased (-0.5%) to 86.025%

Files with Coverage Reduction New Missed Lines %
ext/DynamicPPLForwardDiffExt.jl 1 66.67%
src/logdensityfunction.jl 1 79.41%
src/threadsafe.jl 2 58.47%
src/varinfo.jl 8 84.72%
src/model_utils.jl 10 19.64%
Totals Coverage Status
Change from base Build 12295360310: -0.5%
Covered Lines: 3675
Relevant Lines: 4272

💛 - Coveralls

@penelopeysm penelopeysm force-pushed the py/remove-extra-tests branch from 556cd56 to 696bb3d Compare November 29, 2024 14:17
@TuringLang TuringLang deleted a comment from github-actions bot Nov 29, 2024
@TuringLang TuringLang deleted a comment from github-actions bot Nov 29, 2024
@TuringLang TuringLang deleted a comment from github-actions bot Nov 29, 2024
@TuringLang TuringLang deleted a comment from github-actions bot Nov 29, 2024
@TuringLang TuringLang deleted a comment from github-actions bot Nov 29, 2024
@TuringLang TuringLang deleted a comment from github-actions bot Nov 29, 2024
@penelopeysm penelopeysm force-pushed the py/remove-extra-tests branch from e06591b to 1055a10 Compare November 29, 2024 14:47
test/model_utils.jl Outdated Show resolved Hide resolved
@penelopeysm penelopeysm force-pushed the py/remove-extra-tests branch 5 times, most recently from 21889d7 to 0d7b2d1 Compare December 4, 2024 15:13
@TuringLang TuringLang deleted a comment from github-actions bot Dec 4, 2024
@TuringLang TuringLang deleted a comment from github-actions bot Dec 4, 2024
@TuringLang TuringLang deleted a comment from github-actions bot Dec 4, 2024
@TuringLang TuringLang deleted a comment from github-actions bot Dec 4, 2024
@penelopeysm penelopeysm force-pushed the py/remove-extra-tests branch from 0d7b2d1 to 675b40f Compare December 4, 2024 15:21
@penelopeysm penelopeysm marked this pull request as ready for review December 4, 2024 15:22
@penelopeysm penelopeysm force-pushed the py/remove-extra-tests branch 2 times, most recently from d023c04 to 543a97b Compare December 5, 2024 00:38
@TuringLang TuringLang deleted a comment from github-actions bot Dec 5, 2024
@TuringLang TuringLang deleted a comment from github-actions bot Dec 5, 2024
@TuringLang TuringLang deleted a comment from github-actions bot Dec 5, 2024
@TuringLang TuringLang deleted a comment from github-actions bot Dec 5, 2024
@TuringLang TuringLang deleted a comment from github-actions bot Dec 5, 2024
Copy link

codecov bot commented Dec 5, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 86.02%. Comparing base (0f07520) to head (ea6fd42).

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #733      +/-   ##
==========================================
- Coverage   86.49%   86.02%   -0.47%     
==========================================
  Files          36       36              
  Lines        4272     4272              
==========================================
- Hits         3695     3675      -20     
- Misses        577      597      +20     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Member Author

@penelopeysm penelopeysm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some specific notes on the tests that got removed.

I promise that apart from everything I highlighted here, everything did get moved into the DPPL test suite.

@@ -1,348 +0,0 @@
@testset "compiler.jl" begin
Copy link
Member Author

@penelopeysm penelopeysm Dec 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tests in this file (test/turing/compiler.jl) are the only ones that weren't moved somewhere inside the DynamicPPL test suite. That's because they all actually rely on sampling the posterior and performing numerical checks.

Consequently, I've just moved them to Turing, at least for the time being. This is being tracked at TuringLang/Turing.jl#2419

I don't entirely know the purpose of these tests, but I got quite tired of trawling through all of the tests and figuring out what they were doing / whether they were still needed, so I think this is a reasonable compromise for now.

Comment on lines -166 to -185
@testset "replay" begin
# Generate synthesised data
xs = rand(Normal(0.5, 1), 100)

# Define model
@model function priorsinarray(xs, ::Type{T}=Float64) where {T}
begin
priors = Vector{T}(undef, 2)
priors[1] ~ InverseGamma(2, 3)
priors[2] ~ Normal(0, sqrt(priors[1]))
for i in 1:length(xs)
xs[i] ~ Normal(priors[2], sqrt(priors[1]))
end
priors
end
end

# Sampling
chain = sample(priorsinarray(xs), HMC(0.01, 10), 10)
end
Copy link
Member Author

@penelopeysm penelopeysm Dec 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed this test without replacing it anywhere else. It seems to be checking that we can sample from a model that has its priors in an array. But I think this aspect is already covered in our DEMO_MODELS, specifically demo_dot_assume_observe_index, so I made the call to remove this.

@model function demo_dot_assume_observe_index(
x=[1.5, 2.0], ::Type{TV}=Vector{Float64}
) where {TV}
# `dot_assume` and `observe` with indexing
s = TV(undef, length(x))
s .~ InverseGamma(2, 3)
m = TV(undef, length(x))
m .~ Normal.(0, sqrt.(s))
for i in eachindex(x)
x[i] ~ Normal(m[i], sqrt(s[i]))
end

test/turing/varinfo.jl Show resolved Hide resolved
@penelopeysm penelopeysm force-pushed the py/remove-extra-tests branch from 7bfd233 to 7a93e4b Compare December 8, 2024 20:14
@penelopeysm penelopeysm force-pushed the py/remove-extra-tests branch from 7a93e4b to eafcf3c Compare December 8, 2024 20:32
@penelopeysm penelopeysm requested a review from mhauru December 10, 2024 12:21
docs/src/api.md Show resolved Hide resolved
src/varinfo.jl Show resolved Hide resolved
test/model.jl Outdated Show resolved Hide resolved
test/model_utils.jl Show resolved Hide resolved
test/turing/varinfo.jl Show resolved Hide resolved
test/model.jl Show resolved Hide resolved
@mhauru
Copy link
Member

mhauru commented Dec 12, 2024

What do you think is the right order to merge this and TuringLang/Turing.jl#2419?

@penelopeysm
Copy link
Member Author

What do you think is the right order to merge this and TuringLang/Turing.jl#2419?

This one first, as it's not just tests being modified but also source code so the Turing PR will have to handle the breaking changes here.

Copy link
Member

@mhauru mhauru left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one first, as it's not just tests being modified but also source code so the Turing PR will have to handle the breaking changes here.

Sounds good. Given that this creates a moment when the tests are in neither package, we should make sure to readd them in Turing.jl soon.

Great stuff, thanks @penelopeysm!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Turing integration tests should be moved to (or included in) Turing repo
3 participants