Remove atomic add

[sctiwari]

Need to remove atomic add. Consuming almost 30% of time.

[KenichiNomura]

Do you know the 30% is for what grain size? How many atoms/MPIrank?

[sctiwari]

Total Number of Atom 10752. Program was submitted on 1 KNL nodes with 8 MPI rank and 8 OpenMP thread per rank. No hyper-threading is used.

[KenichiNomura]

So, 1344 atoms/MPIrank with 8 OpenMP threads? I guess the atomic op impacts less on large grain size. It would be nice to investigate it wrt the grain size and # of threads.

EDIT:
I like the idea of removing the atomic ops completely. That'll be beneficial for simulations with small grain size for long time.

[ye-luo]

I think when we go for larger number of threads, the atomic only makes the problem worse.
When we aim large simulations, we tried to keep the number of particles small to reduce time to solution.