-
Notifications
You must be signed in to change notification settings - Fork 146
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How should we understand this function ‘FSE_normalizeM2’? #108
Comments
This is a code I haven't looked at in a while, so I might forget things. If I remember correctly, this function Sometimes, this is relatively straightforward to do, but if the code ever reaches this function, that means the initial distribution of statistics failed and was actually difficult to normalize. It could be for a nb of reasons, a common one being that there are many symbols with a very low yet non-zero probability. Since each symbol needs to receive a frequency of at least 1, they collectively occupy more probability estate than they should. This creates an imbalance, and all other symbols need to make up for this. Now this can be done in multiple ways, with varying degrees of complexity and efficiency. The particular method used here is not important to understand FSE. Any other entropy codec, for example arithmetic coder, will face the same challenge : they need to "normalize" statistics across a clean power of 2 scale. |
Thank you for your reply!Although I still don't quite understand this, I think it's really not important.Maybe in the future, I can understand it.Thank you again |
I am new here and still reading the source code.I can't understand these calculations
https://github.com/Cyan4973/FiniteStateEntropy/blob/dev/lib/fse_compress.c
Line 415~429 in fse_compress.c
The text was updated successfully, but these errors were encountered: