Geometry optimizations on AWS return incorrect structures.

[amyjystad]

Describe the bug
I have an example of the geometry output of a water molecule on
Image

To Reproduce
Steps to reproduce the behaviour:

1. Spin up AWS parallel cluster using the following config file:

Region: us-east-1
Imds:
 ImdsSupport: "v2.0"
Image:
 Os: alinux2
HeadNode:
 InstanceType: m7a.medium
 Imds:
   Secured: true
Scheduling:
 Scheduler: slurm
 SlurmQueues:
 - Name: queue1
   ComputeResources:
   - Name: workers
     InstanceType: c7g.4xlarge
     MinCount: 0
     MaxCount: 20
   Networking:
     PlacementGroup:
       Enabled: true
     SubnetIds:
     - subnet-061b349a7d1667bc8
SharedStorage:
 - MountDir: /shared
   Name: pcluster-fxsl
   StorageType: FsxLustre
   FsxLustreSettings:
     StorageCapacity: 1200
     DataCompressionType: LZ4
     WeeklyMaintenanceStartTime: '7:03:00'
     AutoImportPolicy: NEW_CHANGED_DELETED
     DeletionPolicy: Retain

Install conda
'conda install xtb'
2. Input coordinates: H2O_C0_M1_input.txt

happens with input (include input files)!!
3. 'xtb H2O_C0_M1.xyz --opt > H2O_C0_M1.out'
4. Output:
H2O_C0_M1_output.txt
xtbopt.log

5. Error is not reported in this case (geometry optimization of other structures often fail) But the image in the issue description should suffice to show an issue in the geometry optimization.

Expected behaviour
Image
H2O_C0_M1_expected_output.txt
expected_xtbopt.log

I have other, more exaggerated examples if desired.

[amyjystad]

The 'Expected behavior' are files run locally on my Mac with a conda installed xtb.

[awvwgk]

Since this is a slurm cluster, are you submitting parallel jobs using MPI? In this case xtb will fail because parallelization with MPI is not supported. Instead xtb uses OpenMP for parallelization.

[amyjystad]

No I am submitting with OpenMP. And the same behavior would occur when I ran it from the commandline with only 1 processor.

I spun up a new cluster with instances more similar to my local laptop (Mac with ARM64). And xtb is behaving as expected. This is peculiar behavior but no longer an urgent need to address.

[foxtran]

@amyjystad, your calculation was restarted on AWS. Also, you are using different versions of xtb: it also can be a reason that you do not see this issue on your Mac. Try to run your calc on AWS from scratch. Preferably with xtb 6.7.0.

@awvwgk, which compiler (version + opt.flags) was used for compiling xtb 6.7.1 for Arm? If it is GCC>=13, Arm has a problem :(

[foxtran]

https://conda-metadata-app.streamlit.app/?q=conda-forge%2Flinux-aarch64%2Fxtb-6.7.1-h0fb133d_2.conda

As I see, GCC-13 is used. Unfortunately, I do not see optimization flags, but I would expect -O3 and therefore Arm build is broken. (Fixed in #1121)

[awvwgk]

I guess we can patch the xtb build on conda-forge, either by already including the patch or by reducing the optimization.