[Newcomer] Improve random.repy performance #1388
Assignees
Labels
Comments
choksi81
pushed a commit
that referenced
this issue
May 25, 2014
…and spawning threads until the nodemanager crashes): * The CoordinationAffix, NatDeciderAffix, and NatPunchAffix use cachedadvertise.r2py and advertisepipe.r2py instead of advertise_objects (#1387). This makes it that the continuous Affix readvertisements are done in one central place, thus saving threads; furthermore, our own announcements are immediately available in cachedadvertise. This makes lookups of keys we announce much faster. * Yet more speed in connection setup is gained through the new random_shuffle function (#1388) when selecting a TCP source port (#1362). * The Affixes and the nodemanager now raise errors with more detailed information attached (#1289). * The nodemanager now uses advertisepipe too to announce the node's Zenodotus name to IP address mapping. BE WARNED, node IP changes are currently NOT respected by the logic! A final note: The NAT forwarders currently advertising under '__AFFIX_NAT_FORWARDER__' are incompatible with this release as they are using advertise_object's mangled value format. Before publishing this release, the forwarders need to be upgraded; this release must then be pushed to all existing nodes or they will not be able to contact the (then-upgraded) NAT forwarders. If you want to test this release now and can't wait for upgraded forwarders, there is a single NAT forwarder advertising '__NAT_AFFIX_FORWARDER--post-7209__' that already uses the new format. Change the NAT_FORWARDER_ADVERTISE_KEY definition in your nat_forwarder_common_lib.repy accordingly to reach that special forwarder.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Our functions in random.repy are slow. On my moderately modern !MacBook Pro 8,2:
That doesn't sound terrible at first, but given that we also run on low-power platforms such as Android and !RaspberryPi, the picture changes. Under load, we see runtimes 10 to 200 times as high (for both Repy and Python on a !RasPi). As a consequence, places that use the
random.repyfunctions become prone to timing out. This includes the node manager and Affixes trying to advertise (see r7148, and know that in unrestricted mode, the list of connports has 5000 elements).We should (1) replace our implementation of
random_samplewith a port of Python'srandom.sample, and (2) also port Python'srandom.shufflewhich is quite a bit faster than randomly sampling all of the elements of a list.The text was updated successfully, but these errors were encountered: