You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
#338 added JAVA_HOME auto detection. That's convenient, especially at compile time, since it makes the installation process easier. It also allows Pydoop to work with no JAVA_HOME set at run time, which is also convenient, but it turns out that things can be much slower in that case. Running the entire unit tests suite (minus the avro ones) with no JAVA_HOME is almost 5 times slower. HADOOP_HOME also has an effect, though not nearly as big (a quick comparison on my laptop resulted in 344s with both unset, 75s with JAVA_HOME set and 70s with both set).
Reviewing our caching of these variables (or lack thereof) might help, although not in the case where one is running several Python processes that use Pydoop (auto detection needs to be performed at least once). We do need to document this properly though, so that users make sure they have the most efficient run time setup.
The text was updated successfully, but these errors were encountered:
#338 added
JAVA_HOME
auto detection. That's convenient, especially at compile time, since it makes the installation process easier. It also allows Pydoop to work with noJAVA_HOME
set at run time, which is also convenient, but it turns out that things can be much slower in that case. Running the entire unit tests suite (minus the avro ones) with noJAVA_HOME
is almost 5 times slower.HADOOP_HOME
also has an effect, though not nearly as big (a quick comparison on my laptop resulted in 344s with both unset, 75s withJAVA_HOME
set and 70s with both set).Reviewing our caching of these variables (or lack thereof) might help, although not in the case where one is running several Python processes that use Pydoop (auto detection needs to be performed at least once). We do need to document this properly though, so that users make sure they have the most efficient run time setup.
The text was updated successfully, but these errors were encountered: