Best way to choose from independence tests?

[jjakenichol]

How should I go about choosing a independence test for a dataset? If both parcorr, gpdc, and CMIknn produce graphs with many significant links, how can I evaluate the better model?

It makes sense to use AIC/BIC for hyperparameters such as the alpha significance level, but since each independence test has its own model selection criterion (and CMIknn does not have a model selection criterion?), can they be compared to one another?

[sagarsimha]

You can choose the appropriate independence test based on the assumptions about the distribution. If linear - parcorr, non-linear-gpdc, and CMIKnn is a non parametric test.

[jjakenichol]

Thanks for replying so fast. Right, it's unclear to me how to make those assumptions. I'm working with some climate data, somewhat similar to the data in the Nowak et al. 2020 paper. In there they assumed linearly distributed and used parcorr, but they didn't explain why they made that assumption.

I have some climatologists in consultation and it isn't clear to them either, as far as my data goes. They think there may be both linear and nonlinear relations in the variable set. That would lead me towards gpdc and cmi, since they can find both linear (with lower power) and nonlinear relationships. How should I choose between those two?

CMI seems like the go-to "I just don't know" method, but of course it has its own tradeoffs. Maybe that's ok though?

Does it make sense to run both a linear and a nonlinear test and, to make sure the nonlinear method doesn't miss any linear links, check that it contains the links that parcorr finds?

[jjakenichol]

In Nowak 2020, would it have been wrong to use gpdc? Is it possible to know that from the data or the fitted model, or would you need a subject matter expert to know?

[jakobrunge]

Choosing the right CI test will depend on the assumed underlying functional form of dependencies and also statistical/computational considerations.

ParCorr is linear, GPDC assumes additive (wrt the noise term) nonlinear dependencies and CMIknn makes no assumptions. While that may make CMI look good, the huge(!) trade-off is in statistical detection power of links and computational time. CMI and GPDC (for which a fastGPU-variant GPDCtorch exists) are orders of magnitude slower than ParCorr and will find linear links with much lower detection power. On the other hand, if nonlinearity is present, then ParCorr will yield false positives (not detect absence of link) and false negatives (not detect true link). Therefore, your idea to combine both doesn't work that way.

You can look at scatter plots of variable pairs to see whether strong nonlinearities are present. Small nonlinearities can still be detected with ParCorr (but potentially their effect as a confounder cannot be removed).

[jjakenichol]

Thanks so much. I am running some plots and will see where that leads. I think plotting residuals from a linear model would probably be helpful too.

[jakobrunge]

New tutorial here: https://github.com/jakobrunge/tigramite/blob/master/tutorials/causal_discovery/tigramite_tutorial_conditional_independence_tests.ipynb