effects of activation functions?

[pswpswpsw]

It seems that no one really talk about the effect of activation functions.

Personally, I found Relu gives what the most human-favorable uncertainty result while tanh has too much confidence in the prediction that it does not see.

Similar observation was found by Gal in his paper for dropout as bayesian.

Any thoughts for bayesbyhypernet?

[pawni]

Hi,

Sorry for not replying earlier - somehow I didn't get a notification about that. I have not looked into this but I agree that this is an interesting question to ask :)

I will look into it as soon as I have some time for that - feel free to try it yourself with the code here and share your results.