Trouble using test data in separate file. #147

schiotz · 2022-02-04T09:17:00Z

schiotz
Feb 4, 2022

I would like to test my models with test data in a separate file. In another conversation (#133), @Linux-cpp-lisp pointed me to specifying the file in a separate .yaml file, and use the --dataset-config option, so I run the test with nequip-evaluate --train-dir results/water-Pt-R6-L1-w10/water-Pt-R6-L1-w10 --dataset-config testset.yaml --batch-size 50

Based on the error messages, I have tried to adapt the .yaml file, but now I am stuck. It looks like this:

root: results/water-Pt-R6-L1-w10
run_name: water-Pt-R6-L1-w10

validation_dataset: ase
ase_args:
      format: traj

validation_dataset_file_name: fifth_4_of_5.traj

chemical_symbol_to_type:
  H: 0
  O: 1
  Pt: 2

It looked like I had to use the name validation_dataset instead of dataset, otherwise nequip-evaluate would protest. I also had to add the root line, otherwise I get the error TypeError: __init__() missing 1 required positional argument: 'root'. I assumed I might as well add run_name but did not test it.

The input file is similar to the one used for training, same format, same elements (cut from the same master file). But now I get this error:

Using device: cpu
Loading model... 
loaded model from training session
Loading dataset...
Processing...
Traceback (most recent call last):
  File "/home/niflheim/schiotz/development/NequIP/nequip/nequip/utils/auto_init.py", line 232, in instantiate
    instance = builder(**positional_args, **final_optional_args)
  File "/home/niflheim/schiotz/development/NequIP/nequip/nequip/data/dataset.py", line 785, in __init__
    super().__init__(
  File "/home/niflheim/schiotz/development/NequIP/nequip/nequip/data/dataset.py", line 122, in __init__
    super().__init__(root=root, transform=type_mapper)
  File "/home/niflheim/schiotz/development/NequIP/nequip/nequip/utils/torch_geometric/dataset.py", line 90, in __init__
    self._process()
  File "/home/niflheim/schiotz/development/NequIP/nequip/nequip/utils/torch_geometric/dataset.py", line 175, in _process
    self.process()
  File "/home/niflheim/schiotz/development/NequIP/nequip/nequip/data/dataset.py", line 205, in process
    data = self.get_data()
  File "/home/niflheim/schiotz/development/NequIP/nequip/nequip/data/dataset.py", line 858, in get_data
    [AtomicData.from_ase(atoms=atoms, **kwargs) for atoms in atoms_list],
  File "/home/niflheim/schiotz/development/NequIP/nequip/nequip/data/dataset.py", line 858, in <listcomp>
    [AtomicData.from_ase(atoms=atoms, **kwargs) for atoms in atoms_list],
  File "/home/niflheim/schiotz/development/NequIP/nequip/nequip/data/AtomicData.py", line 392, in from_ase
    return cls.from_points(
  File "/home/niflheim/schiotz/development/NequIP/nequip/nequip/data/AtomicData.py", line 259, in from_points
    raise ValueError("pos and r_max must be given.")
ValueError: pos and r_max must be given.

Maybe I am doing something silly, can anyone point me in the right direction?

Answered by Linux-cpp-lisp

Feb 4, 2022

Hi @schiotz ,

It looked like I had to use the name validation_dataset instead of dataset, otherwise nequip-evaluate would protest.

Can you elaborate on this? That doesn't sound right.

For --dataset-config you must provide all options necessary to define a dataset, which includes root (where to put it), r_max (how to process it), the chemistry (how to translate its types to match the model). While this can involve some copy pasting, we preferred that the user be explicit— silently taking values from the model config could lead to some unpleasant silent bugs, such as mapping the wrong species in the test dataset to the wrong types.

That config.yaml file contains two items called r_max:
d…

View full answer

schiotz · 2022-02-04T09:33:06Z

schiotz
Feb 4, 2022
Author

Follow-up: Shouldn't r_max come from the model specification, be read from results/water-Pt-R6-L1-w10/water-Pt-R6-L1-w10/config.yaml and not be related to the data set?

That config.yaml file contains two items called r_max:

dataset_extra_fixed_fields:
  r_max: 6.0

    [ ... clip clip clip ... ]

r_max: 6.0

Inspired by renaming other dataset_XXX fields, I added

validation_dataset_extra_fixed_fields:
  r_max: 6.0

to testset.yaml, and that appears to make it work. But now I honestly don't think I know what I am doing - how does a separate r_max for the test dataset makes sense? I am confused...

0 replies

Linux-cpp-lisp · 2022-02-04T17:07:07Z

Linux-cpp-lisp
Feb 4, 2022
Maintainer

Hi @schiotz ,

It looked like I had to use the name validation_dataset instead of dataset, otherwise nequip-evaluate would protest.

Can you elaborate on this? That doesn't sound right.

For --dataset-config you must provide all options necessary to define a dataset, which includes root (where to put it), r_max (how to process it), the chemistry (how to translate its types to match the model). While this can involve some copy pasting, we preferred that the user be explicit— silently taking values from the model config could lead to some unpleasant silent bugs, such as mapping the wrong species in the test dataset to the wrong types.

That config.yaml file contains two items called r_max:
dataset_extra_fixed_fields:
  r_max: 6.0

    [ ... clip clip clip ... ]
r_max: 6.0

In this case, the extra dataset_extra_fixed_fields is not necessary (though you won't break anything by computing it) and the default is just to use the "global" r_max specified later.

to testset.yaml, and that appears to make it work. But now I honestly don't think I know what I am doing - how does a separate r_max for the test dataset makes sense? I am confused...

This is correct: in the current stable version you do have to specify r_max in --dataset-config, since it is one of the necessary options to define a dataset, and there are no silent defaults as I said above. That said, I agree that this is a little bit silly when it comes to r_max, and we have actually changed this so it will take the default from the model. That feature just hasn't been released yet.

Thanks!

5 replies

schiotz Feb 5, 2022
Author

Hi @schiotz ,

It looked like I had to use the name validation_dataset instead of dataset, otherwise nequip-evaluate would protest.

Can you elaborate on this? That doesn't sound right.

If I have a .yaml file with database and database_file_name lines, then I get this error:

$ nequip-evaluate --train-dir results/water-Pt-R6-L1-w10/water-Pt-R6-L1-w10 --dataset-config testset_debug.yaml --batch-size 50
Using device: cpu
Loading model... 
loaded model from training session
Loading dataset...
Processing...
Traceback (most recent call last):
  File "/home/niflheim/schiotz/development/NequIP/nequip/nequip/scripts/evaluate.py", line 203, in main
    dataset = dataset_from_config(config, prefix="validation_dataset")
  File "/home/niflheim/schiotz/development/NequIP/nequip/nequip/data/_build.py", line 31, in dataset_from_config
    raise KeyError(f"Dataset with prefix `{prefix}` isn't present in this config!")
KeyError: "Dataset with prefix `validation_dataset` isn't present in this config!"

The error message hinted that I had to rename them validation_database and validation_database_file_name, which seemed to work.

As for your remaining comments: I absolutely agree on the principle of requiring the user to be explicit. I was just not thinking of r_max as a property of the dataset, but as a property of the model, and was therefore surprised. Just because I cannot think of a situation where I would want a different cutoff in the dataset and the model does not mean that it does not make sense :-) - although it does seem a little bit dangerous.

PS. The main conclusion from my experiments is that Nequip works well, and that we will go ahead and use it for a BSc thesis project this spring.

simonbatzner Feb 5, 2022
Maintainer

That's great to hear! :)

Linux-cpp-lisp Feb 5, 2022
Maintainer

If I have a .yaml file with database and database_file_name lines, then I get this error:

This error is a really unfortunate side effect of how the code currently checks for whether validation_dataset or dataset is provided. The real error should be below with a heading line "During handling of the above exception, another occurred:" I'm working to improve this error reporting, added this one to my list!

r_max as a property of the dataset, but as a property of the model, ... although it does seem a little bit dangerous.

Entirely agreed and the new version:

Provides a default r_max taken from the model
Throws an error if you specify an r_max for the dataset different than the model

PS. The main conclusion from my experiments is that Nequip works well, and that we will go ahead and use it for a BSc thesis project this spring.

Great! Glad to hear!

schiotz Feb 7, 2022
Author

This error is a really unfortunate side effect of how the code currently checks for whether validation_dataset or dataset is provided.

I can confirm this. With r_max specified in the testset.yaml file, it works when dataset and dataset_file_name are specified instead of validation_dataset.

Thanks for all your help!

Linux-cpp-lisp Feb 7, 2022
Maintainer

Sure @schiotz , working through these pain points with responsive early users like you helps us get these things right 😄

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Trouble using test data in separate file. #147

{{title}}

Replies: 2 comments 5 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Trouble using test data in separate file. #147

schiotz Feb 4, 2022

Replies: 2 comments · 5 replies

schiotz Feb 4, 2022 Author

Linux-cpp-lisp Feb 4, 2022 Maintainer

schiotz Feb 5, 2022 Author

simonbatzner Feb 5, 2022 Maintainer

Linux-cpp-lisp Feb 5, 2022 Maintainer

schiotz Feb 7, 2022 Author

Linux-cpp-lisp Feb 7, 2022 Maintainer

schiotz
Feb 4, 2022

Replies: 2 comments 5 replies

schiotz
Feb 4, 2022
Author

Linux-cpp-lisp
Feb 4, 2022
Maintainer

schiotz Feb 5, 2022
Author

simonbatzner Feb 5, 2022
Maintainer

Linux-cpp-lisp Feb 5, 2022
Maintainer

schiotz Feb 7, 2022
Author

Linux-cpp-lisp Feb 7, 2022
Maintainer