pyprind with theano, numpy, CUDA, etc.

[chaosmail]

Hi folks,

thanks for your great library, I love it. However, when working with Theano, CUDA or numpy the pyprind progress bar doesn't seem to work (if i remove e.g. the numpy computation it works fine).
What happens: I have to set a manual interval to force an update, but mostly it is not updated. Anyone having similar experience?

Best,
Christoph

[rasbt]

Hi, Christoph,
haven't worked with Theano in a long time, I am using TensorFlow where I use an approach similar to PyPrind working with the sys module, and it works just fine. E.g., I print the "progress" after each epoch (pass over the training set) in the active session, which executes the graph. Here would be an example code snippet of how I implemented this:
https://github.com/rasbt/mlxtend/blob/master/mlxtend/tf_classifier/tf_multilayerperceptron.py#L242

And the actual output in the documentation:
http://rasbt.github.io/mlxtend/user_guide/tf_classifier/TfMultiLayerPerceptron/#example-3-mnist
or an alternative screenshot from the actual IPython notebook:

I haven't tried to print the progress when running on a GPU though, the reason is that my Mac's and MacBooks don't have an Nvidia Card and thus don't support CUDA; whenever I run things on GPU, I have to do it remotely via our HPCC and PBS/Torque, which is why I can't check this locally :(

However, I am curious where you put PyPrind in the Theano code? If it's in the training loop, not in the symbolic graph itself, sth. like

 looping over epochs:
    run compiled theano code
    update pyprind

I believe it should work without problems. However, if it is nested deeper in the parallelized code, I am afraid that this may be an issue of memory sharing during multi-processing, something that may be related to this other joblib issue #21

I'd be happy about further insights and ideas, since making pyprind work for parallel tasks would be a really cool thing!

[chaosmail]

Hi @rasbt

thanks for your very detailed explanation.

Just a quick update from my side, using pyprind and Lasagne + cuDNN works like charm; I cannot explain where this previous error came from.

Best,
Christoph

[rasbt]

Thanks for the update! If I understand correctly, everything works okay now? :) Can we close this issue then?

[chaosmail]

yeah sure ;)