Questions about the details of the dataset preprocessing

[Ikarosy]

Hi, fangwei! I think it quit handy and useful to use 'dvs128_gesture.DVS128Gesture' to load a requested DVS128Gesture dataset as demonstrated in the spikingjelly tutorial, which is terrific. But, I notice that most works don't utilize the whole dvs-videos as samples, instead, they split them into clips of the specific time range as illustrated in your paper, [Incorporating Learnable Membrane Time Constant to Enhance Learning of‘ICCV21]:"Specifically, they illustrate that the time resolution is reduced by accumulating the spike train within every 5 ms and the time range (us) of N-MNIST and CIFAR10- DVS are [290901, 315348] and [1149758, 1459301], re- spectively." So, I wonder if you have done the same thing, because I didn't find any codes in the spikingjelly 0.0.0.0.6 applies this preprocessing nor sutiable interfaces.
Did I miss something? Is it possible to implement the split via spikingjelly? how?
Appreciate your answers!

[fangwei123456]

Hi, the latest SpikingJelly has supported to such a integrating method:

Specifically, they illustrate that the time resolution is reduced by accumulating the spike train within every 5 ms

You can set duration=5000 to implement integrating every 5ms.

Here is a code example:

from spikingjelly.datasets import pad_sequence_collate
from spikingjelly.datasets.dvs128_gesture import DVS128Gesture

train_set = DVS128Gesture(args.data_dir, train=True, data_type='frame', duration=args.duration)
    train_data_loader = DataLoader(
        dataset=train_set,
        collate_fn=pad_sequence_collate,
        batch_size=args.b,
        shuffle=True,
        num_workers=args.j,
        drop_last=True,
        pin_memory=True)

Note that each sample in the DVS dataset has different length, and the integrated frames also have different frames_number, or T. pad_sequence_collate is used to padding frames to the same frames_number. Refer to API doc of pad_sequence_collate and padded_sequence_mask for more details.

[Ikarosy]

Well, by "the clips of specific time range", I mean "how do u sample the frames in the certain range of [290901, 315348] and [1149758, 1459301], respectively in N-MNIST and CIFAR10- DVS ", since both the 'n_mnist.NMNIST' and 'cifar10_dvs.CIFAR10DVS' are the successors of 'sjds.NeuromorphicDatasetFolder', which utilize the the entire dvs-video as the frames of a single sample.

[Ikarosy]

I find the performances of some deep nets are really bad when applied to CIFAR10- DVS. Such being the case, I doubt the lack of
pre-clipping and data augmentation should account for this.

[Ikarosy]

I think it will be very helpful, if the toturial adds some standard data preprocessing methods. Spikingjelly , as a framework, is quite friendly to some rookies like me, though. Appreciate your reply, again!

[fangwei123456]

There are some bugs in API doc. I will fix them in next week.

[fangwei123456]

Well, by "the clips of specific time range", I mean "how do u sample the frames in the certain range of [290901, 315348] and [1149758, 1459301], respectively in N-MNIST and CIFAR10- DVS ", since both the 'n_mnist.NMNIST' and 'cifar10_dvs.CIFAR10DVS' are the successors of 'sjds.NeuromorphicDatasetFolder', which utilize the the entire dvs-video as the frames of a single sample.

In fact, the time range (us) of N-MNIST and CIFAR10- DVS are [290901, 315348] and [1149758, 1459301] means the time range of each sample in N-MNIST and CIFAR10-DVS (the shortest sample in N-MNIST is 290901 (t[-1] - t[0] = 290901), and the longest sample in N-MNIST is 315348), but does not mean selecting 290901 < t < 315348 in the origin events.

I think you mean that we can add a mask [t_min, t_max] to select events whose t satisfies t_min < t_max?

[Ikarosy]

That's exactly what I meant. In the paper, [Temporal-wise Attention Spiking Neural Networks for Event Streams Classification'ICCV21], they have tried some sorts of masks called RCS and IAP on the input event streams to augment the data and save energy and this kind of masks are not rare when preprocessing event streams on the time axis. So, naturally I think spikingjelly should support such kind of preprocessing methods. And, it was my mistaking your words of "the time range (us) of N-MNIST and CIFAR10- DVS are [290901, 315348] and [1149758, 1459301]". Sorry about that. Once again, I appreciate your patience and thank you for reply.

[fangwei123456]

Now SpikingJelly supports for user-defined integrating method: 158235e

https://spikingjelly.readthedocs.io/zh_CN/latest/clock_driven_en/13_neuromorphic_datasets.html#custom-integrating-method