use generators instead of pre-computing all the samples into the memory #31
Conversation
helps to save huge amount of RAM while preserving performance
|
Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. In order for us to review and merge your code, please sign up at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need the corporate CLA signed. If you have received this in error or have any questions, please contact us at [email protected]. Thanks! |
|
Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Facebook open source project. Thanks! |
There was a problem hiding this comment.
@spencebeecher has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
|
Hi @jbinary ! Thanks for the commit! It looks like izip is not supported by python3 (perhaps being automatically handled by zip - https://stackoverflow.com/questions/32659552/importing-izip-from-itertools-module-gives-nameerror-in-python-3-x). Can you run unit tests for python3 and confirm your solution works? I am getting test failure errors around izip for python3 runtime. Thank you! |
|
@jbinary has updated the pull request. Re-import the pull request |
|
ouch. sorry did not realize there are any tests in the first place |
There was a problem hiding this comment.
@spencebeecher has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
|
hey guys, I have no any chance to see what went wrong here, could you give me a clue maybe? |
helps to save huge amount of RAM while preserving performance
It appeared impossible for me to use the lib on a machine with 64Gigs of RAM because it ran out of memory. This simple fix made memory footprint neglectable.