Determining whether issue with VASP interface

[lavroffrob13]

Hi there, I'm attempting to run a CSC calculation across 144 cores, on a rather large unit cell and 3.2 eV energy window. After over a week of running, the calculation never gets past VASP's "entering main loop" statement for a first SCF cycle, which seems like a strange place to hang up (the QMC impurity solver wouldn't be running at this point with VASP, right? Nothing in the output indicates this. The last thing printed is merely the SCF header "N E dE d eps ncg rms rms(c)"). I've made the suggested edits to the VASP source code to use solid_dmft and vasp.lock only say "vasp is running happily." I've also successfully run a single point + LOCPROJ calc with this modified VASP binary outside solid_dmft, using the same INCAR parallelization.

Could this be a solid_dmft issue others have experienced? I wanted to check before I begin putting print statements at various points in the VASP code to see where the hangup is. I've pasted my dmft_config.ini and INCAR below in case something is telling. Thank you in advance for your time!

[general]
seedname = vasp
enforce_off_diag = True
set_rot = hloc

csc = True
plo_cfg = plo.cfg

solver_type = cthyb
n_l = 30

prec_mu = 0.001

h_int_type = kanamori
U = 2.94
J = 0.61
beta = 40

n_iter_dmft_first = 5
n_iter_dmft_per = 2
n_iter_dmft = 20
h5_save_freq = 5

dc = True
dc_type = 1
dc_dmft = True

calc_energies = True

mu_gap_gb2_threshold = 0.05
mu_gap_occ_deviation = 0.4

[solver]
length_cycle = 200
n_warmup_cycles = 10000
n_cycles_tot = 1e+6
imag_threshold = 1e-5
measure_G_l = True
measure_density_matrix = True
measure_pert_order = True

[dft]
n_iter = 4
n_cores = 144
dft_code = vasp
dft_exec = /p/home/lavroff/VASP_DMFT/vasp_std
mpi_env = default
projector_type = plo
store_eigenvals = True

ICHARG = 5 ! 5 for DMFT calculations
ISPIN = 2
ISYM = -1 ! Vasp-PLO CSC doesn't work with symmetries
NELM = 1000 ! to prevent VASP from stopping too early, solid_dmft will stop Vasp
NELMIN = 1000
NELMDL = 0 ! converge the KS wave function first

ISMEAR = 0
SIGMA = 0.02

LMAXMIX = 6
LORBIT = 14
NEDOS = 2001
EMIN = 0.0
EMAX = 3.2
LOCPROJ = 1 : d : Pr

ENCUT = 520

NPAR = 12
KPAR = 12
ALGO = F

[the-hampel]

Dear @lavroffrob13 ,
I believe that I know what problem you encountered. This would be actually a problem with Vasp when using openmpi as mpi backend. Which version of Vasp and MPI are you using?

I made some corrections recently in the Vasp CSC tutorial in solid_dmft: https://github.com/TRIQS/solid_dmft/blob/unstable/doc/tutorials/PrNiO3_csc_vasp_plo_cthyb/2_dmft_csc/dmft_config.ini commenting on this after another user @hmenke reported a similar problem when using Vasp vers >6 together with openmpi 4.x.x.

The problem we see is that with ICHARG=5 activated Vasp hangs at the beginning of the SCF loop like you mentioned. It only prints in the std out until this line appears:

entering main loop
      N       E                     dE             d eps       ncg     rms          rms(c)

and then seemingly does nothing, right? This problem should occur interdependent of running Vasp through solid_dmft or not.

This can be solved by either using an older Vasp version 5.4.4 (in my case I still noticed the problem from time to time), using MPICH as MPI backend in both Vasp and TRIQS, or using Vasp only on one core and leveraging the OpenMP parallelism in Vasp 6.x.

Let's first try to confirm that you indeed run into the same problem.
Best,
Alex

[lavroffrob13]

Also running ICHARG=5 in vasp without using solid_dmft has gotten farther than the main loop entry, seemingly. After half an hour, I have the following:

 found WAVECAR, reading the header
 POSCAR, INCAR and KPOINTS ok, starting setup
 FFT: planning ...
 reading WAVECAR
 the WAVECAR file was read successfully
 charge-density read from file: unknown
 magnetization density read from file 1
 entering main loop
       N       E                     dE             d eps       ncg     rms          rms(c)
DAV:   1    -0.310362005875E+03   -0.31036E+03   -0.94387E+00 22920   0.260E-01
 writing the density matrix to GAMMA

[the-hampel]

I see so then you can run it but it is very slow maybe. Can you maybe cross-check with a smaller system or reduce KPOINT grid and ENCUT? Just to see where the problem is.

[lavroffrob13]

Looks like I observe the same vasp-only problem at only 2 kpoints and ENCUT = 400 eV. I think it's truly stuck. Also, I realized I hadn't added the check for LSUPPRESS_PROJS_EXISTS into vasp (not essential since I'm using PLO but maybe would give a speedup to solid_dmft) so I added it and recompiled. No such luck, unfortunately.

Queue times are pretty short right now on the cluster I'm using, so I can play with NCORE and KPAR and do some trial-and-error. I'm unfortunately having some conda problems on MPICH-compatible clusters, so I haven't been able to try that workaround just yet.

Thank you for all your help!

[lavroffrob13]

Hi Alex, I've gotten past this issue on a different cluster (using Intel MPI) but now solid_dmft seems to execute vasp "n_cores" times in serial instead of just once, which is causing issues with the lock files and (less critically) slowing down the calculation. I'm using "mpi_env = default" and have never had an issue on this cluster executing vasp simply as "mpirun -n x vasp_std". I'm talking with support folks at the cluster, but I figured I'd ask here too, in case it's a known issue with Intel MPI or something. Executing solid_dmft with "-binding" or "-perhost 1" in addition to "-n x" unfortunately doesn't make a difference. This is a SGE cluster, not slurm, so I can't fully follow the example at https://triqs.github.io/solid_dmft/md_notes/run_cluster.html , unfortunately. Please let me know if you'd like me to open a separate discussion post for this.

[the-hampel]

Hi @lavroffrob13 ,
this is usually a sign that the executable is not called with the right MPI environment. In the environment in which you execute triqs/solid_dmft where does which mpirun point to? solid_dmft calls vasp via mpirun, and I believe the problem is that when you call triqs mpirun it points not to the mpi library that was used to compile Vasp but triqs. Please check also which mpi library was used to compile triqs. To use different mpi libraries for vasp and triqs you would have to supply the right mpi executable to solid_dmft. The parameter is not exposed unfortunately.

When executing vasp successfully try to get the full path to the mpi executable with which mpirun then open this line:

solid_dmft/python/solid_dmft/dft_managers/vasp_manager.py

Line 136 in 2e14b41

mpi_exe = mpi_helpers.find_path_to_mpi_command(env_vars, 'mpirun')

wherever you installed solid_dmft. Replace the line with

mpi_exe = /full/path/to/mpi-exe

. Then try again to run via solid_dmft. Now solid_dmft and vasp can have different mpi environments to run with.

Let me know if that works or you need further assistance.