Spatial data prep [BUG] #20
Comments
|
This overflow error occurs because wisdm is trying to convert a value to a fixed-size type and that value is too large to store in the new type. I think what might be happening here is that the template raster has datatype uint8, which only has 8 bytes worth of storage. When the covariate raster is masked to template raster, it's data type is being converted to uint8 as well, but some values (i.e., -9999) are too big to be stored in this data type. Looking at the code for wisdm though, it looks like the templateRaster should consistently be datatype uint16 (which is able to store 16 bytes versus uint8 storing 8 bytes). I think it would be useful to troubleshoot this together when we meet on Friday. Do you have VS Code installed and are you able to install with your privileges? VS Code will help to troubleshoot the issue. Can you also send me the covariate raster for clay? I think it's in the library backup you put on SP but I can't download the backup to my computer because it's too big. |
|
Hi Katie,
I'm trying to coral multiple points you've made here.
This overflow error occurs because wisdm is trying to convert a value to a fixed-size type and that value is too large to store in the new type. I think what might be happening here is that the template raster has datatype uint8, which only has 8 bytes worth of storage. When the covariate raster is masked to template raster, it's data type is being converted to uint8 as well, but some values (i.e., -9999) are too big to be stored in this data type.
Hmm okay I see how data type could be a problem. What is the benefit of constraining the data type of the predictors to match the data type of the template? Maybe it would be better to throw a warning if the datatype of the predictors does not match the template.
Do you have VS Code installed and are you able to install with your privileges?
I don't have VS Code, but I'll install it or request it be installed depending.
Can you also send me the covariate raster for clay?
I've just uploaded the layer separately to the shared drive.
Ok, not sure why the option is greyed out, but I just sent you a new release to test over email that might fix this bug.
I haven't been able to find this, would you mind sending it again?
Please let me know if I've missed anything important. I'm also wondering if you think it would be worth creating a smaller example. I chose this one because of how nicely stepped it already was in terms of occurrence points and geographical spread (we have 5 size stages), but if it's posing so many problems it might be worth me creating an even smaller example.
Best,
Linnea
Linnea Fraser (she/her)
SCC - Biological Science Technician
U.S. Geological Survey
Fort Collins Science Center
2150 Center Ave Bldg C
…________________________________
From: Katie Birchard ***@***.***>
Sent: Wednesday, October 23, 2024 12:04 PM
To: ApexRMS/wisdm ***@***.***>
Cc: Fraser, Linnea (Contractor) ***@***.***>; Author ***@***.***>
Subject: [EXTERNAL] Re: [ApexRMS/wisdm] Spatial data prep [BUG] (Issue #20)
This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding.
This overflow error occurs because wisdm is trying to convert a value to a fixed-size type and that value is too large to store in the new type. I think what might be happening here is that the template raster has datatype uint8, which only has 8 bytes worth of storage. When the covariate raster is masked to template raster, it's data type is being converted to uint8 as well, but some values (i.e., -9999) are too big to be stored in this data type.
Looking at the code for wisdm though, it looks like the templateRaster should consistently be datatype uint16 (which is able to store 16 bytes versus uint8 storing 8 bytes). I think it would be useful to troubleshoot this together when we meet on Friday. Do you have VS Code installed and are you able to install with your privileges? VS Code will help to troubleshoot the issue. Can you also send me the covariate raster for clay? I think it's in the library backup you put on SP but I can't download the backup to my computer because it's too big.
—
Reply to this email directly, view it on GitHub<#20 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BKW2N2UMXBYXBYGP4SMPRSDZ47QLZAVCNFSM6AAAAABPVA6VAOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZTGAZTGOJUGY>.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
|
Thanks for addressing all those points and uploading the clay raster! I think we can discuss more on Friday about the datatype issue, as I'm not entirely sure if I'm correct in where the conversion is happening (it's not explicit in the code). It will become more clear though once we are able to run your library in VS Code. I sent you another email with the ssimpkg file attached. Let me know if you still don't receive it and I'll transfer it a different way. A smaller library would definitely be easier to troubleshoot on. If it's not too much effort and you are able to reproduce the issue with a smaller library that would be super helpful! |
|
I don't see a ssimpkg attachment so its possibly it's blocking the file type from going through. If it's a new release of SyncroSim Studio I would need one of the IT guys to install it. Is it housed online somewhere?
Best,
Linnea
Linnea Fraser (she/her)
SCC - Biological Science Technician
U.S. Geological Survey
Fort Collins Science Center
2150 Center Ave Bldg C
…________________________________
From: Katie Birchard ***@***.***>
Sent: Wednesday, October 23, 2024 4:03 PM
To: ApexRMS/wisdm ***@***.***>
Cc: Fraser, Linnea (Contractor) ***@***.***>; Author ***@***.***>
Subject: [EXTERNAL] Re: [ApexRMS/wisdm] Spatial data prep [BUG] (Issue #20)
This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding.
Thanks for addressing all those points and uploading the clay raster!
I think we can discuss more on Friday about the datatype issue, as I'm not entirely sure if I'm correct in where the conversion is happening (it's not explicit in the code). It will become more clear though once we are able to run your library in VS Code.
I sent you another email with the ssimpkg file attached. Let me know if you still don't receive it and I'll transfer it a different way.
A smaller library would definitely be easier to troubleshoot on. If it's not too much effort and you are able to reproduce the issue with a smaller library that would be super helpful!
—
Reply to this email directly, view it on GitHub<#20 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BKW2N2UT2CY3O3FHRJG4F43Z5AMLFAVCNFSM6AAAAABPVA6VAOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZTGU2DSMJTGI>.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
|
Ok, I put the ssimpkg file in the external sharepoint folder with your library backup and data files. It's not a new release of SyncroSim Studio, just a new release of the wisdm package. |
|
Hi Katie,
Got it! Thanks Katie.
One small user navigation thing: when I go to the Library>General>Packages I can't select the Change button at the bottom. If I click Add it only gives me the option to add a subset of the WISDM versions that I've already added to my packages. It will let me right click the current WISDM package, select change, and then select my new version.
Additionally, I saved the file under my Onedrive rather than on the computer itself and it's giving me access errors. This is just an issue because of my lack of knowledge of our system. However, I'm having trouble getting my library to run on WISDM 2.1.1 in the new location. When I try to delete the WISDM 2.1.1 package from my library it says it's still in use. I then tried to bump back down to a previous WISDM version, but it won't let me. I thought it might then let me delete the 2.1.1 from my OneDrive and put it back from the new location. It also will not let me add an additional 2.1.1 version to my packages. Any ideas how to swap out the OneDrive version for the local one?
Best,
Linnea
Linnea Fraser (she/her)
SCC - Biological Science Technician
U.S. Geological Survey
Fort Collins Science Center
2150 Center Ave Bldg C
…________________________________
From: Katie Birchard ***@***.***>
Sent: Thursday, October 24, 2024 9:09 AM
To: ApexRMS/wisdm ***@***.***>
Cc: Fraser, Linnea (Contractor) ***@***.***>; Author ***@***.***>
Subject: [EXTERNAL] Re: [ApexRMS/wisdm] Spatial data prep [BUG] (Issue #20)
This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding.
Ok, I put the ssimpkg file in the external sharepoint folder with your library backup and data files. It's not a new release of SyncroSim Studio, just a new release of the wisdm package.
—
Reply to this email directly, view it on GitHub<#20 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BKW2N2Q4D7KR4NBEACUZ2W3Z5EESXAVCNFSM6AAAAABPVA6VAOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZVGU3DIMRRGQ>.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
|
That's strange. If you have the wisdm package selected in the Library>General>Packages window, then it should allow you to change the version. Maybe we can take a look at both of your issues tomorrow when we meet. |
|
Okay, sounds good, I just won't be able to test the new version of WISDM on my library before then. I've identified a good data set for a smaller example, but pretty much everything we do uses large rasters. This one will still be the size of Florida, which might be hard to share still.
Linnea Fraser (she/her)
SCC - Biological Science Technician
U.S. Geological Survey
Fort Collins Science Center
2150 Center Ave Bldg C
…________________________________
From: Katie Birchard ***@***.***>
Sent: Thursday, October 24, 2024 1:56 PM
To: ApexRMS/wisdm ***@***.***>
Cc: Fraser, Linnea (Contractor) ***@***.***>; Author ***@***.***>
Subject: [EXTERNAL] Re: [ApexRMS/wisdm] Spatial data prep [BUG] (Issue #20)
This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding.
That's strange. If you have the wisdm package selected in the Library>General>Packages window, then it should allow you to change the version.
Maybe we can take a look at both of your issues tomorrow when we meet.
—
Reply to this email directly, view it on GitHub<#20 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BKW2N2RYID4HHYCLLYRAOMLZ5FGITAVCNFSM6AAAAABPVA6VAOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZWGIZDQOBTHE>.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
I have a raster layer that is throwing a warning during Step 1 that I believe is causing a failure during Step 2. The warning, "Caught exception processing covariate: clay -- <class 'OverflowError'> -- Unable to convert nodata value (-9999.0) to new dtype (uint8)", seems to be related to nodata values. It then seems to not produce the needed layer for Step 2.
To Reproduce
Steps to reproduce the behavior:
I will upload a version of my library for testing. The layer that is causing the error is SG_clay_M
_sl2_100m.tif
Expected behavior
SyncroSim runs spatial data preparation or throws descriptive error during step 1 telling me what format my data needs to be in.
Relevant log output
SyncroSim Version: 3.0.7.0
Operating System: Microsoft Windows NT 6.2.9200.0
Packages:
core -> 3.0.7
wisdm -> 2.0.1
Parent Scenario is: [9] Brassica tournifortii - Data prep
Result scenario is: [35] Brassica tournifortii - Data prep ([9] @ 08-Oct-2024 3:22 PM)
STARTING SIMULATION: 10/8/2024 : 3:22:27 PM
Conda installation: C:\Users\lfraser\AppData\Local\SyncroSim Studio\Conda
Conda environment: C:\Users\lfraser\AppData\Local\SyncroSim Studio\Conda\envs\wisdm-1\wisdm-conda-s3
1 - Prepare Multiprocessing => Total time: 00:05:32
Conda installation: C:\Users\lfraser\AppData\Local\SyncroSim Studio\Conda
Conda environment: C:\Users\lfraser\AppData\Local\SyncroSim Studio\Conda\envs\wisdm-1\wisdm-conda-s3
Processing Covariate: clay (1 of 5) at 15:28:31
Caught exception processing covariate: clay -- <class 'OverflowError'> -- Unable to convert nodata value (-9999.0) to new dtype (uint8).
Processing Covariate: Eta_Oct_Jun (2 of 5) at 15:30:36
Processing Covariate: gHM (3 of 5) at 15:41:09
Processing Covariate: ppt_Oct_Jun (4 of 5) at 15:51:54
Processing Covariate: TMIN_Dec_Feb (5 of 5) at 16:04:54
Saving covariate datasheet at 16:11:47
Done at 16:11:47
2 - Spatial Data Preparation => Total time: 00:43:51
[Failure] The import file was not found: C:\Users\lfraser\OneDrive - DOI\Projects\WISDM\test.ssim.temp\DataTransfer\Scenario-35\SG_clay_M_sl2_100m.tif
SIMULATION COMPLETE: 10/8/2024 : 4:11:51 PM
Total simulation time: 00:49:23
Software versions:
The text was updated successfully, but these errors were encountered: