Bootstrap weights

[alanlujan91]

In this PR, a weight_by argument is added to the estimagic's bootstrap functions, which allows one to weight rows using the survey data.

---

The feature is also know as proportional sampling, which is something we have done in Econ-ARK. See this Wikipedia article.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Files with missing lines	Coverage Δ
src/estimagic/bootstrap.py	99.06% <100.00%> (ø)
src/estimagic/bootstrap_helpers.py	100.00% <100.00%> (ø)
src/estimagic/bootstrap_outcomes.py	100.00% <100.00%> (ø)
src/estimagic/bootstrap_samples.py	100.00% <100.00%> (ø)
src/estimagic/msm_weighting.py	98.27% <ø> (ø)

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests

[timmens]

Hey Alan! Thanks for opening another pull request.

Since this time, the PR is not related to a minor fix but an extension, could you extend the PR description? In particular, could you state the problem (i.e. the missing feature), how this is called in the bootstrap literature, and how you plan to implement it. If you know other libraries that implement this feature, this would also be great to know.

[timmens]

Nice work so far! Thank you!

As mentioned above, I think we need to go through a few things first. If implementing these changes will take too much of your time, you can also re-open this pull request as an issue (feature request). Then we will think about how to implement the feature.

[alanlujan91]

Hey Alan! Thanks for opening another pull request.

Since this time, the PR is not related to a minor fix but an extension, could you extend the PR description? In particular, could you state the problem (i.e. the missing feature), how this is called in the bootstrap literature, and how you plan to implement it. If you know other libraries that implement this feature, this would also be great to know.

Will try to address all of this in the coming days! Thanks!

[timmens]

Hey @alanlujan91, thanks again for working on this feature. We are planning a major refactoring and want to close as many PRs as possible beforehand.

Are you planning to work on this PR in the near future?

[alanlujan91]

hi @timmens

I know this feature as proportional sampling, which is something we have done in Econ-ARK but I don't have a good link to the literature.

https://en.wikipedia.org/wiki/Probability-proportional-to-size_sampling

I can work on this more if you think this is a desired feature, I've just been pretty busy so this hasn't been a priority.

[janosg]

Hi Alan, thanks for the PR. I think this would be nice to have in estimagic. As Tim said, we are planning major refactorings and would like to have few PRs open when we start. Do you think you can finish within the next three weeks? Otherwise it might be a good idea to convert this to an issue and postpone the actual implementation until you have more free time.

[alanlujan91]

@janosg I can definitely finish this within 3 weeks

[alanlujan91]

Modified code according to above comments from @timmens, except for

If the weights are uniform (as in your example), the bootstrap indices with and without the weights argument should be the same. Could you test that? You will need to create two random number generators with the same state.

This one is complicated because of the following note from numpy

"Setting user-specified probabilities through p uses a more general but less efficient sampler than the default. The general sampler produces a different sample than the optimized sampler even if each element of p is 1 / len(a)."

It can be done, but requires generating dummy p's as p = np.ones(n_obs)/n_obs when weights_by is None. Let me know if this is desirable

[timmens]

The changes look really good, thank you! I only have a few minor remarks and two small request, and then we can merge 🎉

Request 1

The function _get_probs_for_bootstrap_indices is not tested yet and the docstring is not that informative. Can you

1. Extend the docstring by explaining what to expect in the different cases (i.e., weight_by is not None, cluster_by is not None, etc.)
2. Write unit tests for these cases. Looking at the function, it is not directly clear to me that the case where both weight_by and cluster_by are not None is correctly handled. With an example in a unit test, this would be much easier to see.

Request 2

It can be done, but requires generating dummy p's as p = np.ones(n_obs)/n_obs when weights_by is None. Let me know if this is desirable

Ah, I see; very good point! In this case, can you add a structural test like this:

def test_get_bootstrap_indices_heterogeneous_weights():
    data = pd.DataFrame({"id": [0, 1], "w_homogenous": [0.5, 0.5], "w_heterogenous": [0.1, 0.9]})
    
    res_homogenous = get_bootstrap_indices(data, weight_by="w_homogenous", n_draws=1_000, rng=get_rng(seed=0))
    res_heterogenous = get_bootstrap_indices(data, weight_by="w_heterogenous", n_draws=1_000, rng=get_rng(seed=0))
    
    # Given the weights, the first sample mean should be close to 0.5, while the second one should be close to 0.9
    assert np.mean(res_homogenous) < 0.75 < np.mean(res_heterogenous)

[janosg]

@alanlujan91 do you think it makes sense to keep this PR open or should we revisit the topic some other time?

[alanlujan91]

Sorry I took so long to get back to this. I finally addressed all of the comments, but now it seems there's some unrelated issue to this PR making the checks fail

[janosg]

Thanks @alanlujan91! Yes, this is unrelated and comes from a new release of DFO-LS. I'll fix it soon.

[timmens]

Hey @alanlujan91,

Sorry for not getting back to you earlier! Thanks for all the changes! I've approved the PR 🎉

It looks very nice now, and with the refactoring, it will be much easier for us to maintain and extend this code section.

Thanks!