Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Share partition cache between selector instances #308

Open
MaxiBoether opened this issue Oct 10, 2023 · 0 comments
Open

Share partition cache between selector instances #308

MaxiBoether opened this issue Oct 10, 2023 · 0 comments
Labels

Comments

@MaxiBoether
Copy link
Contributor

MaxiBoether commented Oct 10, 2023

Currently, each Selector process has their own self._trigger_cache due to the introduction of gRPC using multiprocessing.Processes for true parallelism in #301.

The problem is that we cannot easily use a manager.Dict() here to share a dictionary across processes (leaving performance implications out for now). The problem is that new Selector instances get allocated at runtime, and any Selector instance would need to instantiate a new manager.Dict(). This is not possible as a manager cannot be shared with child classes.

We need to re-think how we do multiprocessing in the Selector for this, probably..

Should be addressed with #178.

#309 has the same root cause, which is requiring access to a parent manager in subclasses.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant