SAXS plugin implementation using electron density #3134

pordyna · 2019-12-12T13:00:54Z

WIP (status 08.05.2020):

Clear up numerical differences between this plugin and an FFT.
Fine tune the test case and finish a testing python script.

This plugin calculates scattering amplitude from a SAXS experiment (Small Angle X-ray Scattering). It can compute the complex amplitude for all charged particles, it is meant to be used with PIC electrons and ions.

It is a continuation of the project started by @ejcjason in #2952 and maintained by @sbastrakov in #2973. The scattering is now calculated from particle density ( bound electron density for ions) that is provided by the particleToGrid algorithm. This simplifies the parallelization compared to the direct calculation from the particle data. Also now the particle shape is taken into account since it's used to derive the averaged field.

What else changed since #2973:

The output is now the complex amplitude and not the intensity anymore. That makes it possible to coherently sum contributions from different species.
It is possible to split the frequency space between devices. The parts of the output array are then passed around and the kernel is executed till all domains have contributed to all frequencies.
The 3rd component in the frequency domain was removed since it can be approximated to 0 in the case of SAXS.
The plugin output is now stored in the openPMD standard. It's written by the SaxsWriter class, using the openPMD API. SaxsWriter is based on @franzpoeschel's CUDA 9.0+, GCC 5.1+, Boost 1.65.1+ #2961.
The output is written in parallel if the frequencies are split between devices.
Arbitrary alignment of the probing beam is now possible. ( Only rotation seems to work correctly for now.)
Transverse and temporal beam profiles are supported. For now just a plane wave and a transversal gaussian, more to come. ( Note, it is useless as long as the beam alignment is not working correctly. )
Amplitude is summed over simulation time steps and not zeroed on each step as before.
There is a new example, which is based on the wakefield example.
There is a saxs.param file now.

The plugin is meant to be used with 3D simulation since otherwise it only calculates a scattering on a two-dimensional sheet. Nevertheless, it compiles and runs in 2D as well.

As mentioned, the beam alignment is not working correctly. It is based on a coordinate transformation from the PIC coordinate system into the comoving system of the probing beam. For now, I could verify that rotation is working properly. Unfortunately, translation is not yet correct. Since a correct cell position in the beam system is needed to obtain the beam profile (shape) at a given cell, for now only a plane wave probe will give correct physical results.

Things that should be still done:

Fixing the coordinate transform.
Writing metadata in the output. Only a few things are written right now.
A python data visualizer class. Now there is only a data reader.

Things that still could be done:

Checkpoint support.
Moving window support. ( I'm not sure if it would make sense to use this plugin with it on.)
More beam profiles, including sth like densityProfiles::FreeFormulaImpl.
Resonant scattering.
Output the probe beam intensity for all simulation cells. This could be useful for verification and debugging and it would help to solve the issue with the coordinate transform.

I had also an idea of generalizing the SaxsWritter into an abstract class for output writing that could be specialized either by inheritance or templates to use in different scenarios, plugins. Could be also used in this plugin for the eventual beam intensity output.

Example output:

At the time I'm only once a week at HZDR and I will not be able to work on the plugin for some time, I'm starting these pull request now, although some features are not giving 100% correct results, to summarize the current status.

@ComputationalRadiationPhysics/picongpu-developers

pordyna · 2019-12-12T14:39:07Z

#2966 passed the compile test and is also using the API so there should be a way to make it work here as well but for now, I don't know how to do it.

include/picongpu/param/saxs.param

pordyna · 2019-12-19T13:22:02Z

I talked with @psychocoderHPC today and we decided that we want to merge it without a 100% correct coordinate transform. I have hidden the functionality that depended on it by moving the settings from the .param file into SaxsProbingBeam.hpp.

I guess this means, it is not a WIP anymore. Can't wait for your reviews 😄

psychocoderHPC

only a few small changes are required but you can wait if someone else is annotating something before you apply the changes.

docs/source/usage/plugins/saxs.rst

include/picongpu/param/saxs.param

psychocoderHPC · 2020-01-07T11:37:57Z

@PrometheusPi @HighIander please review this PR.

sbastrakov

Great work @pordyna ! I only looked through part of the code so far, here are some notes.

include/picongpu/param/saxs.param

docs/source/usage/plugins/saxs.rst

include/picongpu/plugins/saxs/Saxs.hpp

PrometheusPi · 2020-01-16T08:17:41Z

@pordyna Your travis-CI tests fail in the C++ part due to an too old CMake version used.
@psychocoderHPC and @sbastrakov please have a look at this error - perhaps there are some changes needed from the test suite side.

Error message:

$ cmake $TRAVIS_BUILD_DIR/include/pmacc -DALPAKA_ACC_CPU_B_OMP2_T_SEQ_ENABLE=ON

CMake Error at CMakeLists.txt:27 (cmake_minimum_required):
  CMake 3.11.4 or higher is required.  You are running version 3.11.0

-- Configuring incomplete, errors occurred!
The command "cmake $TRAVIS_BUILD_DIR/include/pmacc -DALPAKA_ACC_CPU_B_OMP2_T_SEQ_ENABLE=ON" exited with 1.

$ make -j 2
make: *** No targets specified and no makefile found.  Stop.
The command "make -j 2" exited with 2.
store build cache

Just to be sure, I restarted both failed jobs - but I am pretty sure that that the mismatching version will cause a failure again.

sbastrakov · 2020-01-16T09:36:24Z

@PrometheusPi @pordyna I believe the issue with failing builds is due to cmake version requirements update in #3139 . Rebasing the branch on top of the current dev should solve it.

PrometheusPi

This is a not yet complete review.

I did not go through

Saxs.hpp
Saxs.kernel
SaxsWriter.hpp

In general, the main code looks very good. I made some comments but nothing severe.

The new example file looks like a copy of the LWFA example.
This creates a lot of code and test duplications.
Could you add a SAXS example to an existing example - or did you adjust the param files so significantly, that adding a new exmaple is well motivated. Even if the later is well motivated, there are many param files added that do not differ from the default. Please remove those.

Ideally, you introduce a distinct new SAXS example (suited for testing the SAXS plugin against known analytical solutions) or you add a SAXS compile test to an existing (LWFA) example.

docs/TBG_macros.cfg

docs/source/usage/plugins/saxs.rst

docs/TBG_macros.cfg

docs/source/usage/plugins.rst

share/picongpu/examples/Saxs/include/picongpu/param/saxs.param

share/picongpu/examples/Saxs/include/picongpu/param/species.param

share/picongpu/examples/Saxs/include/picongpu/param/speciesDefinition.param

share/picongpu/examples/Saxs/include/picongpu/param/speciesInitialization.param

share/picongpu/examples/Saxs/include/picongpu/param/starter.param

PrometheusPi

Furthermore, I have the impression that you at two points mix PIC units with SI units, which will result in a (globally) incorrectly scaled result. Relatively, everything should be correct, but the total quantity should be off by a fixed factor. (This is not bad for meter to angstrom, bit for the classical electron radius, since is is not just a 10^10).

@sbastrakov and @psychocoderHPC I checked everything except the SaxsWriter.hpp - this is more in your domain of expertise.

I will perform some run time tests, as soon as the example is clean up.

include/picongpu/plugins/saxs/Saxs.hpp

include/picongpu/plugins/saxs/Saxs.kernel

lib/python/picongpu/plugins/data/saxsReader.py

sbastrakov · 2020-01-22T14:43:13Z

Perhaps, an artificial lattice like used in #2952 provides a better test / demonstration example? There is a setup for PIConGPU in #2973 which, I believe, can be adapted for this plugin.

pordyna · 2020-01-30T15:22:32Z

Alright, going forward 😄 !

I removed the Saxs example for now. I think that a proper example involving some interesting physics could come with an another pull request. For example, one with an artificial lattice, as suggested by @sbastrakov, or maybe a proper simulation with a prestructured target.

There is a 4_saxs.cfg in the LWFA example now. I know that this is not an especially interesting case since we are not interested in SAXS with gas targets. Nevertheless, it is the simplest way to do some run tests since LWFA is easily run in 3D.

I tried to address all of your comments but please let me know if I forgot something.

Fix for the travis style test coming in a moment 😉

sbastrakov · 2020-01-30T15:25:28Z

Great @pordyna ! Thx @PrometheusPi for a detailed review, I would also like to have another look, will put my comments a little later.

sbastrakov · 2020-01-30T15:43:09Z

@pordyna looks like you might have forgotten to rebase your branch on top of the current dev. If so, please do so that all tests are running. Otherwise, please ignore.

pordyna · 2020-01-30T15:45:16Z

@sbastrakov I did rebase it and I just looked into the travis.yml
$HOME/.cache/cmake-3.11.4 is in there. So it must be something else 🤔

pordyna · 2020-01-30T15:52:04Z

@PrometheusPi @psychocoderHPC I have some jupyter notebooks which I used to compare plugin output with python FFT. What is the best way to upload them here?

sbastrakov · 2020-01-30T15:56:55Z

In case these are for the LWFA example with your new .cfg file, I think the lib directory inside this example is best. @psychocoderHPC what is your opinion?

pordyna · 2020-01-30T16:01:42Z

Yes exactly, one for 4_saxs.cfg and an another one for both saxs.param and 4_saxs.cfg modified but in the LWFA example.

pordyna · 2020-06-26T10:40:33Z

It looks like the Access_Type -> Access change in openpmd will come with the next release. Access is used only in the latest documentation not in 0.11.1a. And it doesn't work with 0.11.1a, at least with python.

pordyna · 2020-06-26T11:34:24Z

I tried building docs localy but I get

raise MTimerError('Cannot find file: %s' % os.path.realpath(filename)) breathe.directives.MTimerError: Cannot find file: /home/pawel/Work/PIConGPU-git/picongpu/docs/xml/index.xml

I'm sure I'm just doing something wrong, it used to work before.

Full log:
sphinx-err-a86tokbx.log

PrometheusPi · 2020-07-06T15:49:48Z

@HighIander Could you please comment on @pordyna question regarding the double slit?

PrometheusPi · 2020-07-15T11:53:15Z

@pordyna We just discussed about your pull request in the dev meeting. We will merge and will investigate the discrepancy between theory and simulation later.

PrometheusPi · 2020-07-15T12:00:26Z

@psychocoderHPC Regarding the ci, what should we do?

PrometheusPi · 2020-07-16T09:18:14Z

~~@pordyna I checked the current version of your pull request. The sphinx issue is gone. There are numerous issues/warnings but none related to your SAXS plugin.~~ - I used the wrong branch. Will check again ...

pordyna · 2020-07-28T09:07:00Z

@psychocoderHPC , @PrometheusPi Are we ready to merge, or is there anything left to do ? :)

psychocoderHPC · 2020-07-28T16:30:53Z

I will have a look to this PR tomorrow.

psychocoderHPC · 2020-07-31T20:26:35Z

we will wait until monday for #3316. This will allow that the ci is testing the plugin. The old ci has no openpmd support therefore this PR is not checked.
I wil notify you next week so that you can rebase this pr against dev.

psychocoderHPC · 2020-08-07T15:34:50Z

@pordyna Cou can rebase against the latest development branch. #3316 is merged an OpenPMD is now checked too.

PrometheusPi · 2020-08-11T09:53:20Z

@pordyna I am fine with merging (sorry was on vacation) - please rebase your code.

pordyna · 2020-08-19T07:57:44Z

I see that all 60/60 CI jobs failed 😅 @psychocoderHPC @sbastrakov could you have a look? 😃

sbastrakov · 2020-08-19T08:03:44Z

@pordyna I think there may be a confusion between namespace openPMD in the openPMD API and picongpu::openPMD (which we have in PIConGPU). To work around this, you can use ::openPMD for the openPMD API one in the implementation of your plugin.

Suggested changes from PR reviewers (runtime error on write) -> pmacc::math, GridController access moved rollback change causing runtime error fix axis swap 120x120 -> 128x128 split -> distribute fix wrong density profile adapt thresholds for the 3D case fix wrong dtype in the output after multiplication pylint remove 3.cfg Update docs/source/usage/plugins/xrayScattering.rst Co-authored-by: Richard Pausch <[email protected]> Update docs/source/usage/plugins/xrayScattering.rst Co-authored-by: Richard Pausch <[email protected]> don't use \x0bScalar more info on output in docs fixes for sphinx doc openPMD --> ::openPMD math::Pi -> pmacc:math::Pi

pordyna · 2020-09-03T08:12:54Z

@psychocoderHPC The CI tests passed. I think we are ready to merge. 🚀

pordyna marked this pull request as ready for review December 12, 2019 14:08

pordyna mentioned this pull request Dec 12, 2019

Moving FieldTmpSolvers defintion #3082

Closed

psychocoderHPC changed the title ~~SAXS plugin implementation using electron denisty~~ [WIP] SAXS plugin implementation using electron denisty Dec 13, 2019

psychocoderHPC added the component: plugin in PIConGPU plugin label Dec 13, 2019

psychocoderHPC reviewed Dec 19, 2019

View reviewed changes

include/picongpu/param/saxs.param Outdated Show resolved Hide resolved

pordyna changed the title ~~[WIP] SAXS plugin implementation using electron denisty~~ SAXS plugin implementation using electron denisty Dec 19, 2019

psychocoderHPC requested changes Jan 7, 2020

View reviewed changes

psychocoderHPC requested review from PrometheusPi and HighIander January 7, 2020 11:38

sbastrakov reviewed Jan 7, 2020

View reviewed changes

ax3l changed the title ~~SAXS plugin implementation using electron denisty~~ SAXS plugin implementation using electron density Jan 8, 2020

pordyna mentioned this pull request Jan 9, 2020

Compile time warnings with 0.13.0-dev openPMD/openPMD-api#651

Closed

pordyna force-pushed the topic-saxsPlugin branch from b94a6e3 to ebf9fa9 Compare January 9, 2020 13:20

PrometheusPi requested changes Jan 16, 2020

View reviewed changes

PrometheusPi requested changes Jan 17, 2020

View reviewed changes

ax3l reviewed Jan 18, 2020

View reviewed changes

lib/python/picongpu/plugins/data/saxsReader.py Outdated Show resolved Hide resolved

pordyna force-pushed the topic-saxsPlugin branch from ebf9fa9 to 619efaa Compare January 30, 2020 15:09

pordyna dismissed sbastrakov’s stale review via 74cbb81 June 26, 2020 09:50

pordyna force-pushed the topic-saxsPlugin branch from 95d10d3 to 3c9727b Compare June 26, 2020 11:44

PrometheusPi previously approved these changes Jul 15, 2020

View reviewed changes

PrometheusPi dismissed their stale review via c047752 July 16, 2020 09:48

pordyna force-pushed the topic-saxsPlugin branch 3 times, most recently from 0f87e13 to bc0803d Compare July 20, 2020 10:07

pordyna force-pushed the topic-saxsPlugin branch from bc0803d to 88d9c0e Compare August 14, 2020 07:36

pordyna force-pushed the topic-saxsPlugin branch from 88d9c0e to e69b199 Compare August 19, 2020 12:25

pordyna force-pushed the topic-saxsPlugin branch from e69b199 to b177f35 Compare August 31, 2020 08:54

pordyna and others added 2 commits August 31, 2020 14:41

Implementing SAXS plugin

e8c935b

pordyna force-pushed the topic-saxsPlugin branch from b177f35 to dfe8a99 Compare August 31, 2020 12:42

psychocoderHPC approved these changes Sep 4, 2020

View reviewed changes

psychocoderHPC merged commit 93a6e0e into ComputationalRadiationPhysics:dev Sep 4, 2020

pordyna deleted the topic-saxsPlugin branch September 4, 2020 10:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SAXS plugin implementation using electron density #3134

SAXS plugin implementation using electron density #3134

pordyna commented Dec 12, 2019 •

edited

Loading

pordyna commented Dec 12, 2019

pordyna commented Dec 19, 2019

psychocoderHPC left a comment

psychocoderHPC commented Jan 7, 2020

sbastrakov left a comment

PrometheusPi commented Jan 16, 2020 •

edited

Loading

sbastrakov commented Jan 16, 2020 •

edited

Loading

PrometheusPi left a comment •

edited

Loading

PrometheusPi left a comment

sbastrakov commented Jan 22, 2020

pordyna commented Jan 30, 2020

sbastrakov commented Jan 30, 2020

sbastrakov commented Jan 30, 2020

pordyna commented Jan 30, 2020

pordyna commented Jan 30, 2020

sbastrakov commented Jan 30, 2020

pordyna commented Jan 30, 2020

pordyna commented Jun 26, 2020

pordyna commented Jun 26, 2020

PrometheusPi commented Jul 6, 2020

PrometheusPi commented Jul 15, 2020

PrometheusPi commented Jul 15, 2020

PrometheusPi commented Jul 16, 2020 •

edited

Loading

pordyna commented Jul 28, 2020

psychocoderHPC commented Jul 28, 2020

psychocoderHPC commented Jul 31, 2020

psychocoderHPC commented Aug 7, 2020

PrometheusPi commented Aug 11, 2020

pordyna commented Aug 19, 2020

sbastrakov commented Aug 19, 2020 •

edited

Loading

pordyna commented Sep 3, 2020

SAXS plugin implementation using electron density #3134

SAXS plugin implementation using electron density #3134

Conversation

pordyna commented Dec 12, 2019 • edited Loading

WIP (status 08.05.2020):

What else changed since #2973:

Things that should be still done:

Things that still could be done:

Example output:

pordyna commented Dec 12, 2019

pordyna commented Dec 19, 2019

psychocoderHPC left a comment

Choose a reason for hiding this comment

psychocoderHPC commented Jan 7, 2020

sbastrakov left a comment

Choose a reason for hiding this comment

PrometheusPi commented Jan 16, 2020 • edited Loading

sbastrakov commented Jan 16, 2020 • edited Loading

PrometheusPi left a comment • edited Loading

Choose a reason for hiding this comment

PrometheusPi left a comment

Choose a reason for hiding this comment

sbastrakov commented Jan 22, 2020

pordyna commented Jan 30, 2020

sbastrakov commented Jan 30, 2020

sbastrakov commented Jan 30, 2020

pordyna commented Jan 30, 2020

pordyna commented Jan 30, 2020

sbastrakov commented Jan 30, 2020

pordyna commented Jan 30, 2020

pordyna commented Jun 26, 2020

pordyna commented Jun 26, 2020

PrometheusPi commented Jul 6, 2020

PrometheusPi commented Jul 15, 2020

PrometheusPi commented Jul 15, 2020

PrometheusPi commented Jul 16, 2020 • edited Loading

pordyna commented Jul 28, 2020

psychocoderHPC commented Jul 28, 2020

psychocoderHPC commented Jul 31, 2020

psychocoderHPC commented Aug 7, 2020

PrometheusPi commented Aug 11, 2020

pordyna commented Aug 19, 2020

sbastrakov commented Aug 19, 2020 • edited Loading

pordyna commented Sep 3, 2020

pordyna commented Dec 12, 2019 •

edited

Loading

PrometheusPi commented Jan 16, 2020 •

edited

Loading

sbastrakov commented Jan 16, 2020 •

edited

Loading

PrometheusPi left a comment •

edited

Loading

PrometheusPi commented Jul 16, 2020 •

edited

Loading

sbastrakov commented Aug 19, 2020 •

edited

Loading