Inquiry regarding the impact of the model seed

[keysongkang]

Dear everyone,

I have an inquiry regarding the impact of the model seed. I trained two different models using the same training data and hyperparameters but with different model seeds. I expected some differences between them, but the variance in predicted potential energy appears to be substantial.

When I check the predicted potential energy of one structure using two models, "seed: 0" yields 4.20 eV, whereas "seed: 1" yields 33.68 eV. When comparing the potential energy difference between two distinct structures using two models, "seed: 0" gives 4.17 eV, whereas "seed: 1" gives 1.67 eV.

Below is the input file I utilized.

root: train_0_0/projects
run_name: train
workdir: train
dataset_file_name: ./data-train_0.npz
n_train: 400
n_val: 80
r_max: 5.0
l_max: 3
num_features: 32

seed: 0

#seed: 0                                                   # model seed
dataset_seed: 0                                           # data set seed
requeue: true
append: true                                              # set true if a restarted run should append to the previous log file
default_dtype: float64                                    # type of float to use, e.g. float32 and float64
allow_tf32: false                                         # whether to use TensorFloat32 if it is available
#device: cuda                                             # which device to use. Default: automatically detected cuda or "cpu"

# network
model_builders:
  - SimpleIrrepsConfig
  - EnergyModel
  - PerSpeciesRescale
  - ForceOutput
  - RescaleEnergyEtc

num_layers: 4                                             # number of interaction blocks, we find 3-5 to work best
parity: false                                             # whether to include features with odd mirror parityy; often turning parity off gives equally good results but faster networks, so do consider this

# radial network basis
num_basis: 8                                              # number of basis functions used in the radial basis, 8 usually works best
BesselBasis_trainable: true                               # set true to train the bessel weights
nonlinearity_type: gate                                   # may be 'gate' or 'norm', 'gate' is recommended
resnet: true                                              # set True to make interaction block a resnet-style update
PolynomialCutoff_p: 6                                     # p-value used in polynomial cutoff function
invariant_layers: 2                                       # number of radial layers, we found it important to keep this small, 1 or 2
invariant_neurons: 32                                     # number of hidden neurons in radial function, again keep this small for MD applications, 8 - 32, smaller is faster
avg_num_neighbors: null                                   # number of neighbors to divide by, None => no normalization.
use_sc: true                                              # use self-connection or not, usually gives big improvement

# data set
# the keys used need to be stated at least once in key_mapping, npz_fixed_field_keys or npz_keys
# key_mapping is used to map the key in the npz file to the NequIP default values (see data/_key.py)
# all arrays are expected to have the shape of (nframe, natom) except the fixed fields
dataset: npz                                              # type of data set, can be npz or ase
#dataset_file_name: ./nequip.train.npz                     # path to data set file

key_mapping:
  z: atomic_numbers                                       # atomic species, integers
  E: total_energy                                         # total potential eneriges to train to
  F: forces                                               # atomic forces to train to
  R: pos                                                  # raw atomic positions
  CELL: cell
  PBC: pbc
#  stress: stress
chemical_symbols:
  - In
  - Na
  - O

# logging
wandb: false                                                                       

verbose: info                                                                      
log_batch_freq: 5                                                                  
log_epoch_freq: 1                                                                  

# scalar nonlinearities to use — available options are silu, ssp (shifted softplus), tanh, and abs.
# Different nonlinearities are specified for e (even) and o (odd) parity;
# note that only tanh and abs are correct for o (odd parity).
nonlinearity_scalars:
  e: silu
  o: tanh

nonlinearity_gates:
  e: silu
  o: tanh

# training
learning_rate: 0.01                                       # learning rate, we found 0.01 to work best - this is often one of the most important hyperparameters to tune
batch_size: 1                                             # batch size, we found it important to keep this small for most applications 
max_epochs: 10000                                         # stop training after _ number of epochs
train_val_split: random                                   # can be random or sequential. if sequential, first n_train elements are training, next n_val are val, else random 
shuffle: true                                             # If true, the data loader will shuffle the data
metrics_key: validation_loss                              # metrics used for scheduling and saving best model. Options: loss, or anything that appears in the validation batch step header, such as f_mae, f_rmse, e_mae, e_rmse
use_ema: True                                             # if true, use exponential moving average on weights for val/test
ema_decay: 0.999                                          # ema weight, commonly set to 0.999
ema_use_num_updates: True                                 # whether to use number of updates when computing averages

# loss function
# different weights to use in a weighted loss functions
loss_coeffs:
 forces: 1
 total_energy:                                                                    
    - 1
    - PerAtomMSELoss
# stress: 1
 
# output metrics
# early stopping based on metrics values. 
# LR, wall and any keys printed in the log file can be used. 
# The key can start with Training or validation. If not defined, the validation value will be used.
early_stopping_patiences:                                                          
  validation_loss: 50

early_stopping_delta:                                                             
  validation_loss: 0.005

early_stopping_cumulative_delta: false                                             

early_stopping_lower_bounds:                                                       
  LR: 1.0e-5

early_stopping_upper_bounds:                                                       
  wall: 1.0e+100

metrics_components:
  - - forces                              
    - rmse                                                  
    - report_per_component: True
    
  - - total_energy
    - rmse
    - PerAtom: True

# the name `optimizer_name`is case sensitive
optimizer_name: Adam                          
optimizer_amsgrad: true
optimizer_betas: !!python/tuple
  - 0.9
  - 0.999
optimizer_eps: 1.0e-08
optimizer_weight_decay: 0

# lr scheduler
lr_scheduler_name: ReduceLROnPlateau
lr_scheduler_patience: 50
lr_scheduler_factor: 0.5

#shift and scale the data
per_species_rescale_shifts: dataset_per_atom_total_energy_mean
per_species_rescale_scales: dataset_forces_rms

Is this expected? I'm uncertain about the reason behind this occurrence.
If anyone has insightful comments, please share! Your input would be greatly appreciated.
Thank you in advance.