xnet

[brucefan1983]

usage in nep.in:

use_xnet # change to use the Cauchy activation function instead of tanh

Original activation in the paper is $f(x) = \frac{\lambda_1 x + \lambda_2}{x^2 + d^2}$, where $\lambda_1$, $\lambda_2$, and $d$ are trainable parameters.

I modified it to $f(x) = \frac{\lambda_1 x + \lambda_2}{x^2 + d^2 + 0.01}$ to avoid singularity. Is this necessary?

Ref: Cauchy activation function and XNet

Does not seem to be useful, will give up soon...

[BBBuZHIDAO]

It's glad that there is a new activation function and you have achieved it so fast 🤩. But I have a small question for the modification of the denominator. It means d in non-modified equation must be bigger than 1. This setting will loss some functions, whose peak value is closer to 0.
I try to draw function $\frac{x}{x^2+d^2}$, $\lambda_1=1$, for different d. Here is the result:

Although $\lambda$ will scale the function, the inflexion point message nay loss.

[brucefan1983]

So far the feedbacks are negative :(