Parity plot

[saeedshahi]

Hi,
I want to plot a parity plot for the prediction of energies and dft energies and I know should I use nequip-evaluate but I'm confused about output.xyz file. Which energies are predictions and which one are dft (original) energies?
Do you have any python code that I can use to plot the parity plot?

Best,
Saeed

[Linux-cpp-lisp]

Hi @saeedshahi ,

nequip-evaluate outputs only the predictions of the model. If you want to plot them alongside your truth labels in a parity plot, each frame in output.xyz will contain a field original_dataset_index indicating the index of the frame in the original dataset that the model was evaluated on. From this, you can load that original frame to get the ground truth energies and forces.

Please note that while output.xyz is currently sequential there is no guarantee that it will be in the future, which is why you should use original_dataset_index.

[sinaDFT]

Hello @Linux-cpp-lisp,

I am trying to find original_dataset_index in the output.xyz file which is produced using nequip-evaluate .... output.xyz includes Atoms objects which include "symbols", "pbc", "cell", "energies","forces" and "calculator" and I cannot find the original_dataset_index field in the output.xyz file. I need to have the diagram of parity plot to see how predicted energies and forces are wrt ground truth data. Please let me know how I can find the correct indexes of the test set that has been used for prediction.

Thanks,
Sina

[Hongyu-yu]

I make some changes in evaluate.py for parity plot and error plot. Hope it helps. If demanded, I would like to submit a PR.
evaluate.zip

[Linux-cpp-lisp]

@sinaDFT Hm, it should always be output: https://github.com/mir-group/nequip/blob/main/nequip/scripts/evaluate.py#L378-L380

What version are you on?

[Linux-cpp-lisp]

@Hongyu-yu ah nice, thank you for posting this here!

I think the best solution long-term, since this seems to be a request from a lot of people, is to add an --output-fields-from-dataset option like --output-fields that adds those fields into the output XYZ file with some prefix, i.e.:

nequip-evaluate --train-dir ../whatever/dir --output test.xyz --output-fields total_energy,forces --output-fields-from-dataset total_energy,forces

Then text.xyz would have keys forces and total_energy from the model's predictions, but also keys original_dataset_forces and original_dataset_total_energy. This can easily be done at lines https://github.com/mir-group/nequip/blob/main/nequip/scripts/evaluate.py#L378-L380.

Then parity plotting, error histograms, etc. can be done by users on this one XYZ file.

I'd be more than happy to have examples of making those plots from your code in the examples/ directory; I just don't want to include plotting code in the nequip-evaluate command because everyone has their own preferred way to make plots, sometimes you want the data as part of a bigger plot, etc.

[Hongyu-yu]

Hi @Linux-cpp-lisp ,
I get your point and agree with you :)

[sinaDFT]

@Hongyu-yu Thank you for this. In oder to use this evaluate.py, should I install the nequip package in a editable fashion using pip install -e nequip or I need to do something else in order to make effect of this change in the package?

[Hongyu-yu]

Hi @sinaDFT ,
You can just run it with python evaluate.py --train-dir ./ and make your own changes to the plot.

[sinaDFT]

@Linux-cpp-lisp I understand your point about the existence of original_dataset_index in the evaluate.py code, but using the following command:
nequip-evaluate --train-dir ./results/Structure/Train_dir/ --batch-size 25 --output output.xyz --output-fields original_dataset_index
I am not getting the indexes of the test set data. Please let me know if I am missing something here. By the way, I am using the nequip __version__ = "0.5.6".

Regarding those changes that you mentioned, e.g. adding --output-fields-from-dataset to the evaluate.py, should I install the nequip package in the editable fashion as I need to make changes to the codes in the package or there is other ways to do it? please let me know. Thanks.

[Linux-cpp-lisp]

I tried quickly on both main (0.5.6) and develop this:

nequip-train configs/minimal.yaml
nequip-evaluate --train-dir results/aspirin/minimal/ --output test.xyz --output-fields original_dataset_index --batch-size 25
head -n 3 test.xyz 

and get

21
Properties=species:S:1:pos:R:3:forces:R:3:energies:R:1 original_dataset_index=0 energy=-405630.28125 pbc="F F F"
C        2.13448882      -0.98436105      -0.19521789     -28.33660889     -19.13697624      37.66880417  -19318.93945312

which has original_dataset_index. Can you confirm that this works on your system?

Regarding those changes that you mentioned, e.g. adding --output-fields-from-dataset to the evaluate.py, should I install the nequip package in the editable fashion as I need to make changes to the codes in the package or there is other ways to do it? please let me know. Thanks.

Yes, if you are planning to make edits to the code I recommend using pip install -e. If you are interested in contributing a PR to the code, there is information on our development process here: https://github.com/Linux-cpp-lisp/nequip/blob/optimize-command/CONTRIBUTING.md.

[sinaDFT]

@Linux-cpp-lisp Thank you for trying this. I was reading the output.xyz file using ase.io.read() and for some reasons, it did not print out the original_dataset_index for me. I tried it with simply loading file line by line like what you did and I can see that the original_dataset_index is there indeed.

[Linux-cpp-lisp]

Ah good to hear! Yes, if you read it with ase.io.read it should be in the atoms.info dict.

[Linux-cpp-lisp]

Hi @sinaDFT @Hongyu-yu

I've implemented the --output-fields-from-original-dataset field as discussed above (ead9c5d) and included an example script for using it to make a parity plot (cb3a347).

[Hongyu-yu]

That's great! Thanks for the work and acknowledgement:)

[sinaDFT]

Hi @Linux-cpp-lisp @Hongyu-yu

Thank you for the implementation and details. I just want your suggestion to how to use the changed scripts. Should I reinstall the nequip in an editable mode or some other ways to make it effective with these changes?

[Linux-cpp-lisp]

These are new features on the develop branch, which contains a number of breaking changes. You can either create a new environment downloading and installing the develop branch (in editable mode) and train new models or wait for the merge of the current develop branch into a release to do the same.

(In general new releases shouldn't require training new models, but this update involves a number of breaking changes)

[sinaDFT]

@Linux-cpp-lisp Thank you.