Unstable training of LSTMSNN

[ghost]

Hi there!

I'm a PhD student working on time series prediction with SNN and have been using your LSTM-SNN with little success so far. In the case of learning to predict a 1D sine signal, using exactly the same code as with a successful LSTM, sending the signal real values directly to the model as input, I can't get a successful prediction and rather get divergent outputs while predicting the sine entirely. The predictor uses (x(t), c(t)) to predict x(t+1). Have you been facing such issues?
I've attached a test prediction trial below.
Thank you.
YC

[fangwei123456]

Hi, the first thing is to update your SpikingJelly to the lastet version (install from source codes from github) because there was a bug (#101) in old version.

Then if your network still does not converge, you can change the surrogate function from the defauly binary activation to the continuous activation: sigmoid for surrogate_function1 and tanh for surrogate_function2.

spikingjelly/spikingjelly/activation_based/rnn.py

Line 498 in 19b0688

surrogate_function1=surrogate.Erf(), surrogate_function2=None):

Then this Spiking LSTM should behavior identical to LSTM. You can check if your network converges after using sigmoid and tanh.

[ghost]

Hi, thank you so much for your reply.
Indeed I updated your package with the source code and now trainings are stable. But the results without changing the surrogate functions are still not really satisfying (see picture attached). I use an Adam optimizer with a scheduler on the learning rate from 1e-2 to 1e-4 each 30 epochs for a total of 100 epochs.
In activation_based.surrogate I have found the Sigmoid class but not the Tanh, where can I find it?

Thank you.
YC

[fangwei123456]

In activation_based.surrogate I have found the Sigmoid class but not the Tanh, where can I find it?

In activation_based.surrogate, all functions are binary during forward (outputs are 0/1). You can try to use torch.nn.Sigmoid and torch.nn.Tanh, which will outpout float values and make your Spiking LSTM become LSTM. By doing this and comparing with LSTM, you can check how many "accuracy loss" you get if you use binary activation.

[ghost]

Thank you, I tried this but got the following error: "AttributeError: 'Sigmoid' object has no attribute 'spiking'"

[fangwei123456]

Hi, this is caused by 'assert self.surrogate_function1.spiking == self.surrogate_function2.spiking'. You can avoid this by:

import torch
import torch.nn as nn
from spikingjelly.activation_based import rnn

Sigmoid = nn.Sigmoid()
Sigmoid.spiking = True
Tanh = nn.Tanh()
Tanh.spiking = True
net = rnn.SpikingLSTM(input_size=1, hidden_size=2, num_layers=3, surrogate_function1=Sigmoid, surrogate_function2=Tanh)

print(net(torch.rand([1, 1])))

[fangwei123456]

However, your spiking LSTM can converge now. So, you may not need this method.

[ghost]

Ok perfect thank you now with this setting it works just as a normal LSTM.

[fangwei123456]

It seems that your current result is acceptable. I suggest that you can add num_layers and hidden_size to inhance the fitting ability of your Spiking LSTM and check if you can get higher accuracy.

[ghost]

Ok so I've been playing over num_layers, hidden_size and the number of epochs but hardly can get better predictions while increasing the complexity of the model. It seems that the SNNLSTM needs many more epochs than LSTM to converge. Sometimes while increasing the num_layers, I get a constant or nearly constant prediction.
Below are two examples, in order:
First one hyperparameters: num_layer=1, hidden_size=10, epochs=300, scheduler(start=1e-2, step=100, gamma=0.1)
Second one hyperparameters: num_layer=1, hidden_size=60, epochs=300, scheduler(start=1e-2, step=100, gamma=0.1)
While the LSTM predicts it perfectly with num_layer=1, hidden_size=10, epochs=20
Are these results consistent with yours?

[fangwei123456]

I have not used spiking LSTM for actual tasks. You can try to load weights from LSTM to your spiking LSTM. This method works well in SNN.