Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: release GIL for most "run_block_on" calls #388

Merged
merged 1 commit into from
Mar 11, 2024

Conversation

dariogoetz
Copy link
Contributor

This PR wraps most run_block_on calls in a py.allow_threads function, releasing the Python GIL for the duration of the run_block_on call. This is especially important for PartitionConsumerStream.next which might block indefinitely.

The issue arose with me, when running a PartitionConsumer in a separate thread and using the stream method. The underlying call to PartitionConsumerStream.next was waiting for incoming messages while holding the Python GIL, effectively prohibiting the main thread to do any work.

I think that releasing the GIL is mostly important for the PartitionConsumerStream.next method, because it may block indefinitely (if no messages arrive), but I wrapped all other run_block_on calls as well, while I was at it.

I left out the instances of run_block_on in the context of Cloud and CloudAuth as I have not worked with those, yet.

@digikata
Copy link
Contributor

Thanks for the PR @dariogoetz, we appreciate you taking the time to submit it. We'll queue this up to review and test it right away!

@digikata
Copy link
Contributor

digikata commented Mar 11, 2024

Looks good to me, added an issue to expand further to other areas
#390

Thanks again @dariogoetz

@digikata digikata merged commit b0a8652 into infinyon:main Mar 11, 2024
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants