Bitrounding + Lossless compression #3599
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| using BitInformation | ||
|
|
||
| struct BitRounder{K} | ||
| keepbits :: K | ||
| end | ||
|
|
||
| # number of keepbits (mantissa bits) for each variable | ||
| default_bit_rounding(::Val{name}) where name = 23 # single precision default | ||
| default_bit_rounding(::Val{:u}) = 2 | ||
| default_bit_rounding(::Val{:v}) = 2 | ||
| default_bit_rounding(::Val{:w}) = 2 | ||
| default_bit_rounding(::Val{:T}) = 7 | ||
| default_bit_rounding(::Val{:S}) = 16 # 12 at the surface, 16 deep ocean | ||
| default_bit_rounding(::Val{:η}) = 6 | ||
|
|
||
| function BitRounding(outputs = nothing; | ||
| user_rounding...) | ||
|
Comment on lines
+16
to
+17
There was a problem hiding this comment. For the purpose of figuring out good defaults perhaps we should include There was a problem hiding this comment. Then |
||
|
|
||
| keepbits = Dict() | ||
|
|
||
| # TODO: | ||
| # Check that the dimensions of keepbits are | ||
| # compatible with outputs if user_rounding | ||
| # contains an abstract array (support functions?) | ||
|
|
||
| for name in keys(outputs) | ||
| if name ∈ keys(user_rounding) | ||
| keepbits[name] = user_rounding[name] | ||
| else | ||
| keepbits[name] = default_bit_rounding(Val(name)) | ||
| end | ||
| end | ||
|
|
||
| return BitRounding(keepbits) | ||
| end | ||
|
|
||
| # Getindex to allow indexing a BitRounder as | ||
| Base.getindex(bit_rounding::BitRounding, name::Symbol) = BitRounding(bit_rounding[name]) | ||
|
|
||
| function round_data!(output_array, bit_rounder::BitRounder) | ||
|
|
||
| # The actual rounding... | ||
| keepbits = bit_rounder.keepbits | ||
|
|
||
| # TODO: make sure that the rounding happens | ||
| # as we expect (priority to the vertical direction!) | ||
| round!(output_array, keepbits) | ||
|
|
||
| return output_array | ||
| end | ||
|
|
||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is interesting. Why is there a difference between T, S? Is this specific to the simulation that this was tested on, or can we be sure this is valid for all simulations, past climates, future climates, idealized simulations at other resolutions, etc?
It seems we need to have default bit rounding for passive tracers.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Although relatively robust through time and space, this depends on a lot of things, also whether your unit carries some offset around (e.g. Kelvin vs ˚C, density vs density anomaly). So it's tricky to generalise. I suggest to have some reasonable defaults if someone uses bit rounding (default
nothingor single precision as you like) but suggest to highlight that this should be checked similar to how I did it here with the bitinformation analysis above.For global ocean simulations I expect these to be reasonable defaults. I believe for now this is mostly to reduce the filesizes for OMIP simulations
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
True, I'm just not sure that OMIP is going to be the most common use case, so there's a question about what default is appropriate here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The OMIP defaults might belong in the
ClimaOceansetup, perhapsThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could set the defaults here as 23 mantissa bits (=Float32 precision, whether you use Float32 or 64) and then lower in ClimaOcean?