independent km caches for distinct cases; clear the Aesara cache if it's large #1127
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Put a checksum into the `KmcountsCached` cache filename so different cases get independent cache files, e.g. when switching git branches, Parca options during parameter optimization, or mono/polycistronic operons. This renames the cache file from `fixtures/endo_km/km3.cPickle` to `parca-km-1918837868.cPickle`, for instance. Q. Does anyone prefer the "fixtures" directory name? The cache files `cache/parca-km-*.cPickle` will accumulate until `make clean`. Does this succeed in distinguishing current cases? We could make this more sensitive by checksumming more inputs or less picky by rounding `Kmcounts.astype(np.float16)`. See #1123
This `make clean` will clear the current Aesara cache if it's larger than a threshold. 30MB seems fair the cache is 11MB locally and 350 MB in `~tahorst` on Sherlock. Aesara's cache management is not straightforward, partly to handle multiple processes and NFS.
tahorst
approved these changes
Jul 23, 2021
There was a problem hiding this comment.
Looks great! This should be a very helpful option!
This should also fix #996 - kmcounts accounts for the first 3 args to the loss function calculation but to be completely sure we don't need to recalculate, maybe we should add isEndoRnase and alpha to the checksum calculation and explicitly include the other inputs to the loss function (totalEndoRnaseCapacity, rnaConc and degradationRates)?
1/2 of code review feedback: add isEndoRnase and alpha to the cache checksum calculation to account for all the loss function's inputs.
This should fix #996 by putting more cross-checks in the Km counts cache, but I don't have a test case for that. Ideally we'd factor out this part of the very long `setKmCooperativeEndoRNonLinearRNAdecay()` function and add test cases for it. Q. Should `arrays_differ()` set the absolute and/or relative tolerances? Adding to the cache file grew it from 37K to 110K. That should be fine but
|
Nice! I think that should work. I don't know exactly when the issue in #996 was caused but since we're now checking everything it should be good to go. Thanks for tacking on the extra work on this PR! Only thing to note is a typo: |
... which affects the cache file content so it's esp. good to fix before merging into master.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For Parca for operons #1123 : Put a checksum into the
KmcountsCachedcache filename so different cases get independent cache files, e.g. for mono/polycistronic operons or switching git branches. This succeeded with Parca options in a parameter optimization run. It's limited by needing to compute the checksum before readingKmcountsCachedbut it's still backed up by the content check.It could be made more picky by checksumming more inputs or less picky by rounding
Kmcounts.astype(np.float16).This renames the cache file from
fixtures/endo_km/km3.cPickleto (e.g.)cache/parca-km-1918837868.cPickle.The cache files will accumulate until
make clean.Q. Does anyone prefer the
fixtures/endo_km/directory name?Fix Can Aesara cache be cleared automatically? #1120 :
make cleanwill clear the current Aesara cache if it's larger than a threshold -- 30MB seems fair since the cache is 11MB locally and 350 MB in~tahorston Sherlock.Aesara's cache management is not straightforward, partly to handle multiple processes and NFS.