Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Debug flaky CI integration tests #127

Closed
wants to merge 23 commits into from
Closed

Conversation

lukasschwab
Copy link
Owner

@lukasschwab lukasschwab commented Oct 14, 2023

Description

CI tests are failing for latest commit on master. I'm unable to reproduce locally. Debugging.

Breaking changes

List any changes that break the API usage supported on master.

Relevant issues

List GitHub issues relevant to this change.

Unexpected test failures:

Checklist

  • (If appropriate) README.md example usage has been updated.

@lukasschwab lukasschwab changed the title Use setup-python v4, cache pip deps Standardize on HTTPS Oct 14, 2023
@lukasschwab
Copy link
Owner Author

Current theory: flakiness is caused by rate limiting.

@lukasschwab lukasschwab changed the title Standardize on HTTPS Debug flaky CI integration tests Oct 14, 2023
Should consolidate into a single test suite class.
@lukasschwab
Copy link
Owner Author

Even with all these changes (slowing things waaay down) the build is flaky.

I'm curious: can I up the parallelism again?

@lukasschwab lukasschwab force-pushed the debug-failing-ci branch 3 times, most recently from 0140a5e to 591e490 Compare October 15, 2023 00:23
@lukasschwab
Copy link
Owner Author

lukasschwab commented Oct 15, 2023

This conclusion sucks, but I think I'm trying to adapt the tests to accommodate temporarily flaky API behavior. As I've poked around, the behavior has been unstable (e.g. earlier only HTTPS worked; now only HTTP works).

It's possible this is driven by arXiv is tightening their rate limiter behavior... but that only explains this if they enforce the rate limit through a variety of unclear responses (TCP hangup vs. empty result set vs...).

I might consider simplifying this PR to two fixes:

  1. Hard-pinning the feedparser version.
  2. Consolidating to HTTPS.

@lukasschwab
Copy link
Owner Author

3e55127 might've just been lucky.

@lukasschwab lukasschwab mentioned this pull request Oct 15, 2023
1 task
@lukasschwab
Copy link
Owner Author

Closing; PRs in the v2.0.0 release handle the best changes explored here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant