Latest Velox 3.12 .emd files are not readable due to major changes in file format

[Philipp-Muller]

Velox 3.12 introduced changes in the .emd file format. The new files are not readable anymore.

https://www.fei-software-center.com/tem-apps/velox-download-page/

By now I can confirm that both STEM images and EDS elemental mappings are affected. I therefore assume it is a general issue.

I can provide example files within the next few days if required.

Thanks for the continuous support and development!

[ericpre]

Yes, that would be great if you could provide some files. Ideally, the smallest possible file so that they can be added to the test suite - if you are happy with that of course! 😃

[Philipp-Muller]

Disclaimer: The following files were acquired with velox 3.10 and then opend with Velox 3.12 (offline) this triggers the conversion to the new format.

[Philipp-Muller]

Au SI+Spectrum+Linescan 155 kx.zip
several testfiles.zip
Au SI 155 kx.zip

[ericpre]

It seems that the group "Operations" is missing from the file that you shared and it is used to parse information about detector/elemental map. This can be skipped but then we don't know what elemental map it is... or the type of DPC images, etc.

@robindegen, by any chance, are you still working on Thermo Fisher Velox EMD? If so, could you please advise on what happen to the "Operations" group? 😃

[Philipp-Muller]

If you require specific support from Thermo-Fisher please post here. The Velox developers follow the discussion.

[shoving-tfs]

If you require specific support from Thermo-Fisher please post here. The Velox developers follow the discussion.

Indeed we are following the discussion, Velox 3.12 introduced a file format change to support upcoming features.

Just for understanding, we call the 'pre 3.12 file format' the old file format. And the file format from Velox 3.12 and higher the new file format.

Important to understand
The new file format only applies to Spectrum Imaging (SIFeature).

Here is a quick (incompleet) summary of the changes.

What changed

• The file format version in \Version is now 11.
• The old file format 'operations' are replaced by a more data driven philosophy.
• The old file format 'operation settings' are now moved to \SharedProperties.
• The old file format operation connections is now removed, in favor of a data reference model.
• In the old file format \Features linked to \Operations, in the new file format \Features link to \SharedProperties\DataReference.
• In the old file format operations linked to other operations and to data. In the new file file format \Features link to everything.

Feel free to ask any question.

[ericpre]

Thank you @shoving-tfs for the information!
Can you advice on how to get the information about the images, for example, the element for EDS maps, or operation between detector quadrant for DPC data? I had a quick look at the new structure but couldn't find it what I was after and it would much more simple if you could provide some poiners.

[shoving-tfs]

Hi @ericpre thank you for your question, as I have started before only SIFeature has changed, DPC has not been effected.

Regarding the element for EDS Maps:

1. /Displays/ImageDisplay contains the list of all the image displays. A EDS Map is just an image display.
2. These entries contain a json encoded dictionary that contains 'data', 'id', 'settings' and 'title'.
3. The 'id' is the name of the element. 'data' is pointing to the data reference in SharedProperties/ImageSeriesDataReference/<UUID> which in turn is pointing to the /Data/Image/<UUID> where the image data is located.

I hope this is enough to get you started.

[ericpre]

Thank you @shoving-tfs, that's exactly what was necessary! 😃
I was asking about DPC, because currently the Operations group is used to get information to know the type of DPC. However, from reading the file provided above, the code to read the label of EDS maps seems to be working well for DPC images in version 11.

@shoving-tfs, @Philipp-Muller, would you please be able to provide small test file to add the test suite in this repository? The files above are standard in size but this is too large to add to the repository, because we want to avoid the repository getting unnecessarily large.

This should be fixed in #232 and we just need to add small test files.

[ericpre]

@shoving-tfs, @Philipp-Muller, I added some detailed explanation on the test data files in #233. See documentation output at https://rosettasciio--233.org.readthedocs.build/en/233/contributing.html#making-test-data-files.
Any feedback (something unclear, ambiguous, etc) on this section is more than welcome!

[ThorstenBASF]

Attached you will find some EMD files for testing. Smallest size for imaging is 256x256. As last time, the files are acquired with Velox 3.10 and converted with 3.12.
You also find a double converted BCF-File (BCD>EMD>EMD). Till now this file type is not supported, the error said something about a "prune" format.
Velox_3.12_testfiles.zip

Thanks for your support!

[ericpre]

Thank you @ThorstenBASF for sharing the files. Is the "Test SI 16x16 ReducedData 215 kx.emd" file also one of these pruned file?

With the "Test BCF SI 64x64 converted", there is indeed a warning saying the spectrum image can't be read because "pruned files" are not supported and the warning suggest the user not to prune the file! ;) This is expected.

[Philipp-Muller]

The pruning is not a user decision. This is the file format which is generated by velox with the Bruker .bcf to .emd converter. "Test BCF SI 64x64 converted" is the only file of this type.

"Test SI 16x16 ReducedData 215 kx.emd" means that the file size reduction of velox was used. after that you cannot trim the individual frames of the eds acquisition. This is a user decision and could be avoided.

[ericpre]

Okay, then it sounds like this is saving the data in the same way. An alternative to the conversion is to load the bcf file directly using hyperspy. In any case, it would be good to improve the warning.

For reference, here is the original discussion for the prune files: hyperspy/hyperspy#1662 (comment).