How to speed the sampling up on windows

[TeaforTwo1011]

Hi team,

I am running a simulation study, where I simulated 100 participants (variable: Sub), 5 different conditions (variable: S), 30 trials per condition for each participants, and assume a hierarchical structure for all the parameters.

The model graph looks like this:

Then I started sampling with this line of code, let the .sample() decide the specific sampler to use:
infer_model = model_ddm.sample(chains=3, cores=1, draws=2000, tune=1000)

And this is what I got:

It took already 19hrs to draw 1790 samples from chain 0 and I really do not know whether it is normal speed and what can I do to speed it up.

It would be really helpful if I can get some suggestions from your side.

Many thanks!

[AlexanderFengler]

Hi @TeaforTwo1011,

sorry for the delayed response on this.

I would do two sanity check here.

1. Run a much smaller round of sampling and check if results look reasonable at all. It is possible that the chains just got stuck. We worked hard on resolving chain initialization issues and made some good progress reflected in the latest release (and the main branch of the git repo).

2. Try a subset of ~ 25 subject and see if results come out reasonable in that case.

Lastly, to narrow in on what might be the reason behind the slowdown, can you provide the printout after importing hssm? Does this mention something about 'using numpy c as backend for blas'?

[HNStaggs]

Hello. I am also experiencing a super long run time. My sample is 770 people and its going on 13 hours. I am running a hierarchical model with 3 trials per participant. When I was initially setting up the environment, I was getting errors about blas and numpy. To set-up my environment, I had to use numpy 1.25.2 and pymc 5.11. I am just wondering if this is normal? The nuts sampler is using 4 chains in 4 jobs and it says 0 divergences so far. I am going to try your suggestion running a smaller sample and also specifying the sampler to use ('nuts_numpyro' and 'nuts_blackjax') to see if this improves.

[zaizibai]

3 trials per participant seems not a good choice for SSM/DDM. See Lerche, V., Voss, A., & Nagler, M. (2017). How many trials are required for parameter estimation in diffusion modeling? A comparison of different optimization criteria. Behavior Research Methods, 49(2), 513-537. https://doi.org/10.3758/s13428-016-0740-2

[AlexanderFengler]

Hi @HNStaggs,

we will release 0.2.2 soon, which includes a few improvements concerning robustness for cases such as yours. In your case, the sampler simply stopped [this can happen] or did it finally finish?

But I do second @zaizibai concerning whether your experimental data is a-priori a good fit for the types of models we provide through HSSM. 3 trials is very very few, and essentially all your parameter estimation would come through shrinkage at the group level.

Make sure to use noncentered=True in your case as the non-centered parameterization is exactly suitable for a situation where you have very little information per subjects (atom).

[HNStaggs]

Thank you both for the feedback. I am looking forward to exploring the new version. I ended up looking at my sample in groups based on our outcome variable, with the underlying groups being matched for sex and age. I ran DDM on the groups separately so the number of group trials is over 300 per group. I was following the logic of this paper: Comparing younger to older (Ratcliff, R., Thapar, A., McKoon, G.A. Diffusion model analysis of the effects of aging on brightness discrimination. Percept Psychophys. 65, 523-535; 10.3758/bf03194580 (2003)). I found statistically significant differences in the group parameter phenotypes using this method. However, in the future, we are updating our task to have more trials because using the hierarchical model is preferred.