Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scale & offset factor not considered #131

Closed
milankl opened this issue Dec 4, 2020 · 2 comments
Closed

Scale & offset factor not considered #131

milankl opened this issue Dec 4, 2020 · 2 comments

Comments

@milankl
Copy link
Contributor

milankl commented Dec 4, 2020

I have a nc file that was converted from grib, which used the linear integer quantisation method:

##### Attributes #####

Variable            Name                Value                                   
--------------------------------------------------------------------------------
global              history             2020-12-04 14:31:15 GMT by grib_to_net..
global              Conventions         CF-1.6                                  
latitude            units               degrees_north                           
latitude            long_name           latitude                                
time                units               hours since 1900-01-01 00:00:00.0       
time                calendar            gregorian                               
time                long_name           time                                    
longitude           units               degrees_east                            
longitude           long_name           longitude                               
no2                 missing_value       -32767                                  
no2                 units               kg kg**-1                               
no2                 add_offset          2.568999213205925e-7                    
no2                 long_name           Nitrogen dioxide                        
no2                 scale_factor        7.840441656613366e-12                   
no2                 _FillValue          -32767

julia> no2
900×451×29 Array{Int16,3}:
[:, :, 1] =
 -32766  -32766  -32766  -32766  -32766  …  -32766  -32766  -32766  -32766
...

such that the Int16-values in this array are masked for -32767, and then scale linearly from add_offset in scale_factor steps to add_offset+(2^15-1)*scale_factor. I like the idea to be able to access the actually stored data in Int16 but maybe an automatic conversion to Float32/64 could be an option too?

@visr
Copy link
Member

visr commented Dec 4, 2020

Indeed it's not being applied right now. I'll close this in favor of #39. There is also some more discussion here:
JuliaGeo/NCDatasets.jl#57 (comment), about how to implement this in a type stable manner.

@visr visr closed this as completed Dec 4, 2020
@meggart
Copy link
Member

meggart commented Dec 8, 2020

Another option would be to wrap your NcVar into a CFDiskArray from DiskArraytools (see here: https://github.com/meggart/DiskArrayTools.jl/blob/6728512ed9c33e1fa261bf38b2de3b97589ce310/src/DiskArrayTools.jl#L155) This uses broadcast into the output array and therefore looping over chunks, avoiding to make a copy of the full array. Note also that, since this is all based on DiskArrays now, also the operation:

v = NetCDF.open(filename, varname)
v_scaled = v .* scale_factor .+ offset

will be lazy and the scaling only applied when you actually read the data and v_scaled will have the correct element type.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants