Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

softplus is not correct #580

Closed
Dr-Chen-Xiaoyu opened this issue Jan 3, 2024 · 2 comments
Closed

softplus is not correct #580

Dr-Chen-Xiaoyu opened this issue Jan 3, 2024 · 2 comments
Labels
bug Something isn't working

Comments

@Dr-Chen-Xiaoyu
Copy link

Hi, brainpy team:

Based on the source code (https://brainpy.readthedocs.io/en/latest/_modules/brainpy/_src/math/activations.html#softplus):

def softplus(x, beta=1, threshold=20):
  x = x.value if isinstance(x, Array) else x
  return jnp.where(x > threshold, x * beta, 1 / beta * jnp.logaddexp(beta * x, 0))

The softplus activation in brainpy is not correct in the reverted linear part, whose slop should always be 1. The behavior can be replicated with codes:

import matplotlib.pyplot as plt
import brainpy as bp
import brainpy.math as bm
softplus=bp.dnn.Softplus(beta=0.1) # try different beta
scale = 5e1
x=bm.linspace(-scale,scale,20001)
plt.plot(x,softplus(x))

image

Based on equation of softplus (https://en.wikipedia.org/wiki/Rectifier_(neural_networks)) and usage of threshold (https://pytorch.org/docs/stable/generated/torch.nn.Softplus.html#torch.nn.Softplus), It might be corrected by this:

def softplus(x, beta=1, threshold=20):
  x = x.value if isinstance(x, Array) else x
  return jnp.where( x > threshold/beta , x, 1 / beta * jnp.logaddexp(beta * x, 0))

or might just directly use jax's softplus, if it can be auto-graded by brainpy:

def softplus(x, beta=1):
  return jax.nn.softplus(beta*x)/beta

Best,
XiaoyuChen, SJTU

@Dr-Chen-Xiaoyu Dr-Chen-Xiaoyu added the bug Something isn't working label Jan 3, 2024
@chaoming0625
Copy link
Collaborator

Thanks for the report~

@chaoming0625
Copy link
Collaborator

chaoming0625 commented Jan 3, 2024

Please see #581.

Moreover, I have increased the default threshold to 40.

Currently, the softplus function is:

Figure_1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants