Deploying to Speck2fDevKit

Introduction

This guide will show you how to deploy your pretrained model to a SynSense Speck2fDevkit with a SPECK^TM neuromorphic system-on-chip processor for both simulated event streams and real-time inferencing. This assumes you have already Trained and evaluted your own dataset, please follow this guide if you have not already done so.

Since this is specific for the Speck2fDevKit, no other SynSense boards will work. The callbacks for almost all of the functions for on-chip inferencing in this repo are specific for the Speck2fDevKit so must be modified to your board should you have a different model. Note though, we do not guarantee that it will work with any previous or future board models.

We provide this guide from the view of our paper which considers an 80x80 region of interest crop of the DVS and uses a final 10x10 image. Whilst we could expand on how to modify the ROI parameters to handle larger or smaller input, we will keep everything as default purely from a visual place recognition (VPR) perspective. If you want to deviate from this, please refer to the samna documentation to make the necessary edits to the samna configurations in the run_speck.py script. We are happy to try and provide support where possible through submission of an issue.

Getting started

With your board plugged in, please ensure you have set the necessary USB permissions for the Speck2fDevKit.

If you are wanting to use the online inferencing system, you will also need to install samna which is not included in our conda-forge package:

# Install samna into your python environment
pip install samna
# Check the importation of samna works
python
> import samna

Note for samna, your Python version must not be greater than 3.11 which as of writing this guide is the highest version supported.

Evaluate your model on-chip using simulated event streams

Once everything is installed and you've checked it imports, we can now take a pretrained model and test its performance on chip. To do so, we'll add the --simulated_speck flag which tells LENS to generate simualted event streams, convert them into SPECK events, and send them to the Speck2fDevKit:

 python main.py \ 
--dataset <your dataset> \
--camera <your camera> \
--reference <your reference> \
--reference_places <number of reference places> \
--query <your query> \
--query_places <number of query places> \
--matching \
--simulated_speck

The model will collect the number of spikes generated in the output layer and generate a similarity matrix for each reference against every query.

Running the online inferencing model

To run the online inferencing model, we need to parse the --event_driven flag. We no longer need the --matching flag since we perform online sequence matching, which we can evaluate after inferencing:

# Run the online inferencing model with Speck2fDevKit
python main.py \ 
--dataset <your dataset> \
--camera <your camera> \
--reference <your reference> \
--reference_places <number of reference places> \
--query <your query> \
--query_places <number of query places> \
--event_driven

This will spawn a samnagui GUI instance that will show you the input event stream and the power consumption for the 5 power tracks (IO, logic, RAM, VDDD, & VDDA) measured at 20 Hz. It is possible to suppress this GUI by parsing the --headless flag:

# Run the online inferencing in headless mode
python main.py \ 
--dataset <your dataset> \
--camera <your camera> \
--reference <your reference> \
--reference_places <number of reference places> \
--query <your query> \
--query_places <number of query places> \
--event_driven \
--headless

This can reduce a bit of the computational overhead if running on a compute resource limited machine, however note that running it this way will prevent you from obtaining power metrics from the inferencing model.

Output from the online inferencing model

During online inferencing, there are several outputs that are collected throughout and afterwards.

Output spikes

During inferencing, the model will be saving the output spikes to ./lens/output/<output folder>/spike_data.npy during each --timebin value, which sets how many milliseconds to collect output spikes for. In general, this value should match how many milliseconds you generated your event frames for. For example, if you collected events over 1 second you would set:

# Collect output spikes over 1 second
python main.py \ 
--dataset <your dataset> \
--camera <your camera> \
--reference <your reference> \
--reference_places <number of reference places> \
--query <your query> \
--query_places <number of query places> \
--event_driven \
--timebin 1000

Similarity matrix

During inferencing, online sequence matching is performed every time the number of output spike vectors == the --sequence_match argument value. This forms a similarity matrix you can use for downstream precision analysis, saved to ./lens/output/<output folder>/similarity_matrix.npy. By default, --sequence_match is set to 4, therefore every 4s a sequence match will occur.

Power measurements

Power is continually monitored throughout the inferencing. When the samnagui window process is closed, the inferencing model is stopped and the power is collected from the relevant power_sink and saved to ./lens/output/<output folder>/power_data.npy.

If using --headless, you will not be able to get this power data since collecting and saving it follows the destroying of the GUI window which obviously does not spawn if this argument is called.

Input events

If you wish to output the event stream for every --timebin, we can do so by parsing the --save_input flag which will save NumPy arrays of events to ./lens/output/<output folder>/events/:

# Save the event stream during inferencing
python main.py \ 
--dataset <your dataset> \
--camera <your camera> \
--reference <your reference> \
--reference_places <number of reference places> \
--query <your query> \
--query_places <number of query places> \
--event_driven \
--save_input

Note, this can be computationally taxing for compute restrained platforms if you're collecting a lot of events which can make the inferencing model stumble on occasion. If you want to rebuild the events into event frames, you can use the ./lens/tools/manual_eventframe_generator.py script to convert SPECK events into .png images.

Create a dataset from the DevKit

More than likely, you will want to collect a dataset from the Speck2fDevKit to train a model for online inferencing. We provide an easy and convenient script that will automatically collect events, generate images, and a necessary .csv file for training your model.

To collect data you need to parse the --collect_data argument. Use the --timebin argument to specify over how many milliseconds to collect events for. You can name your traversal using the --data_name argument as well, outputting images into the relevant dataset and camera subfolder:

# Collect data from SPECK
python main.py \ 
--dataset <your dataset> \
--camera <your camera> \
--collect_data \
--data_name <your data name> \
--timebin 1000 # Collect over 1 second, for example

Limitations

For deployment on SPECK^TM, models must be no bigger than 144kB in total (see Memory Constraints and Network Sizing. If customizing the network training for on-chip deployment, input size, feature layer size multiplier, and number of learned places must be carefully considered. Generally speaking, a neuron model size of 100 input, 200 feature, and 80 output neurons is roughly the maximum sizing. These constraints do not impact off-chip deployment.

Troubleshooting & issues

If you're experiencing issues with using the Speck2fDevKit, please raise an issue.