Algorithms in Nengo: Spiking data

Algorithms in Nengo: Spiking data#

Before you start, be sure to Download and save this file and this file to your 495_workspace folder.

(Some of the) Code Cred

This tutorial covers the following:

  • Format of spiking camera data

  • How to read spiking camera data

  • How to run spiking camera data through Nengo neurons

Motivation#

Many times we process spiking data into regular images, then perform image processing (such as detection and classification) using traditional CPUs. This is a valid way to do things!

BUT WHAT IF we instead feed the spiking data straight into neurons?? We can then just perform the image processing with neurons as well, further reducing power consumption and keeping the dimensionality of data reduced to events.

Resources#

If you are interested in using spiking data for your final project, take a look at a bunch of options here, here, and here – or we can ask Dr. York to let us collect some data.

This website consolidates a lot of spiking-data resources.

This survey paper discusses different event-based cameras available and how they are processed in sections 2.5 and 3.1 respectively.

Set up#

Ensure you are using your 495 Virtual Environment before you begin!

Then, import Nengo, and other supporting libraries into your program to get started:

import numpy as np
import matplotlib.pyplot as plt
from IPython.display import HTML
from matplotlib.animation import ArtistAnimation

import nengo

import pandas as pd

Load your data (in the correct format)#

Hopefully you recall our discussion on event-based data earlier in the semester. If not, take a look back at Lecture 6.

Hopefully you also recall building and utilizing Address Event Representation during our Path Planning lab. Combine those two concepts and you have spiking-camera data.

We have spiking-camera data containing pixel-coordinates with a timestamp of an event occuring along with it’s polarity (0 or 1) from the Physics department as well as from Dr. York. This data is stored in a *.csv file. We will read the *.csv file and convert it to an input array using csv_to_event_array, we will then use the array as our input using a Nengo node that calls readSpikeData.

The *.csv files contain an array for every event consisting of the following values:

  • x: The horizontal coordinate of the event.

  • y: The vertical coordinate of the event.

  • p: The polarity of the event (0 for off, 1 for on).

  • t: The event timestamp in microseconds.

The readSpikeData function will send 1’s as input at each time step at which there is an event. Each x-y location (i.e. pixel) will connect to a neuron. There will be both a positive and negative neuron for each pixel.

Other fun things about data
We can also generate data if we don’t have our own (copy code from here). AND there are even Github repositories like this one out there where you can take any data you’d like and convert it to spiking data like this.

def csv_to_event_array(csv_filename: str, start_frame: int, end_frame: int) -> np.ndarray:
    # ======= DVS camera - Physics dept ===========
    df = pd.read_csv(csv_filename, names=['x', 'y', 'p', 't'])
    
    # set initial time to 0 for Nengo simulator to run data right away
    sub_df = df[(start_frame <= df['t']) & (df['t'] <= end_frame)]

    sub_df['t'] = sub_df['t'] - sub_df['t'].iloc[0]

    events_list = [(y, x, p, t) for x, y, p, t in sub_df.values]
    events_array = np.array(events_list, dtype=[('y', 'i4'), ('x', 'i4'), ('p', 'i4'), ('t', 'i4')])

    return events_array

# the csv file we will convert and the time range of our events

# ======= DVS camera - Physics dept ===========
csv_filename = 'City_Data.csv'
start_time = 585861
end_time = 685846
# =============================================

# ======== Prophesee camera - ECE dept ========
"""csv_filename = '10msLED.csv'
start_time = 0.99e6
end_time = 1.07e6"""
# =============================================

# Our events array we will feed in through a Nengo node
events = csv_to_event_array(csv_filename, start_time, end_time)
print("Successfully read %r" % csv_filename)

View the data (NOT using neurons yet)#

This is all just a lot of python-ness to display a video. Big takeaway - we are adding up events over a window of time. Our events occur over the course of 100ms. We take 10ms chunks and set any pixels within that 100ms with an event to 1 (or -1 should we have a polarity of 0). This gives us 10 frames.

You can read all about animations here.

# ======= DVS camera - Physics dept ===========
img_height = 180
img_width = 240
t_length_us = end_time - start_time #value is in microseconds
t_length_s = t_length_us * 1e-6 #convert prior value to seconds
dt_frame_us = (10 * 1e-3) * 1e6 #this value is also in microseconds
t_frames = dt_frame_us * np.arange(int(round(t_length_us / dt_frame_us))) #number of frames in video
# =============================================

# ======== Prophesee camera - ECE dept ========
"""img_height = 720
img_width = 1280
t_length_us = end_time - start_time #value is in microseconds
t_length_s = t_length_us * 1e-6 #convert prior value to seconds
dt_frame_us = (10 * 1e-3) * 1e6 #this value is also in microseconds
t_frames = dt_frame_us * np.arange(int(round(t_length_us / dt_frame_us))) #number of frames in video"""
# =============================================

fig = plt.figure()
imgs = []
for t_frame in t_frames:
    t0_us = t_frame
    t1_us = t0_us + dt_frame_us
    t = events[:]["t"]
    m = (t >= t0_us) & (t < t1_us)
    events_m = events[m]

    # Empty frame
    frame_img = np.zeros((img_height, img_width))

    for sub_event in events_m:
        # show "off" (0) events as -1 and "on" (1) events as +1
        event_sign = 2.0 * sub_event["p"] - 1
        frame_img[sub_event["y"], sub_event["x"]] = frame_img[sub_event["y"], sub_event["x"]] + event_sign
    
    img = plt.imshow(frame_img[:, ::-1], vmin=-1, vmax=1, animated=True)
    imgs.append([img])

ani = ArtistAnimation(fig, imgs, interval=50, blit=True)
HTML(ani.to_jshtml())

Read Spike Data function to input into Nengo#

We can now load our data into a Nengo model using the readSpikeData function we will create. Input arguments to this function are:

  • data:The path of the file to read from. Can be a .aedat or .events file. Format of the file will be detected from the file extension.

  • pool: Number of pixels to pool over in the vertical (first argument) and horizontal directions (second argument), respectively. The larger the pool, the fewer neurons required and the faster things run. Note: image indices must be evenly divisible by pool size in respective directions.

  • img_height and image_width: Dimensions of the camera data.

Note that we can have positive (leading edge) and negative (trailing edge) spikes from our camera called the polarity of the event. In this case we have 2: one for positive events, one for negative. The first half of our neurons represent positive events, the second half negative.

# We'll make a simple object to implement the delayed connection
class readSpikeData:
    def __init__(self, event_data, img_height, img_width, pool=(1,1)):
        self.xvals = event_data[:]["x"]
        self.yvals = event_data[:]["y"]
        self.time = event_data[:]["t"] 
        self.pol = event_data[:]["p"]
        self.img_ht = img_height
        self.img_wt = img_width
        self.pool = pool

    def step(self, t):

        dt = .001
        t_lower = (t - dt) * 1e6
        t_upper = t * 1e6

        times = self.time
        indices = np.nonzero((times >= t_lower) & (times < t_upper))[0]

        pool_y, pool_x = self.pool
        
        data = np.zeros((self.img_ht*self.img_wt*2,), dtype=int)
        for index in indices:
            if self.pol[index] == 1:
                # Manually flatten data using (i*y_len)+j **note x,y vals swapped
                data[self.yvals[index]*self.img_wt + self.xvals[index]] = 1/dt 
            else:
                data[self.img_ht*self.img_wt + self.yvals[index]*self.img_wt + self.xvals[index]] = 1/dt
        
        # reshape the data so pooling computations are more intuitive
        data_sz = data.shape
        pos_data = data[0:int(data_sz[0]/2)]
        neg_data = data[int(data_sz[0]/2)::]
        pos_data = pos_data.reshape(self.img_ht, self.img_wt)
        neg_data = neg_data.reshape(self.img_ht, self.img_wt)

        if pool_x > 1 or pool_y > 1:
            pooled_ht = int(self.img_ht/pool_y)
            pooled_wt = int(self.img_wt/pool_x)
            pooled_posdata = np.zeros((pooled_ht,pooled_wt))
            pooled_negdata = np.zeros((pooled_ht,pooled_wt))

            for i in range(0, self.img_ht, pool_y):
                for j in range(0, self.img_wt, pool_x):  
                    pooled_posdata[int(i/pool_y),int(j/pool_x)] = np.mean(pos_data[i:i+pool_y, j:j+pool_x])
                    pooled_negdata[int(i/pool_y),int(j/pool_x)] = np.mean(neg_data[i:i+pool_y, j:j+pool_x])
            
            pooled_posdata = pooled_posdata.reshape(pooled_ht* pooled_wt)
            pooled_negdata = pooled_negdata.reshape(pooled_ht* pooled_wt)

            pooled_data = np.append(pooled_posdata, pooled_negdata)
            return pooled_data
        else:
            return data

Build your model#

This should all look familiar! We are feeding in our input values (i.e. our spike data) through a node, representing the pixel changes with neurons, and reading the data out using probes.

# ======= DVS camera - Physics dept ===========
pool = (1, 1) #(pool_height y direction, pool_width x direction)
    # NOTE // image indices must be evenly divisible by pool size in respective directions!

# ======== Prophesee camera - ECE dept ========
# pool = (10, 10)

inp = readSpikeData(events, img_height, img_width, pool)

model = nengo.Network(label="Spiking Data")
with model:
    input_node = nengo.Node(inp.step)

    input_neurons = nengo.Ensemble(int(img_height/pool[0] * img_width/pool[1] * 2), 1)

    nengo.Connection(input_node, input_neurons.neurons, transform=1.0)# / np.prod(pool))

    probes_nodes = nengo.Probe(input_node)
    probes = nengo.Probe(input_neurons.neurons)

with nengo.Simulator(model) as sim:
    sim.run(t_length_s)

View the data (using Nengo neurons!)#

This looks very much the same as viewing the data with regular Python - except now we are reading spikes from our probed neurons!

sim_t = sim.trange()
shape = (len(sim_t), int(img_height/pool[0]), int(img_width/pool[1]))

output_spikes_pos = sim.data[probes][:,0:int(img_height/pool[0])*int(img_width/pool[1])].reshape(shape) * sim.dt
output_spikes_neg = sim.data[probes][:,int(img_height/pool[0])*int(img_width/pool[1]):int(img_height/pool[0])*int(img_width/pool[1])*2].reshape(shape) * sim.dt

dt_frame = dt_frame_us * 1e-6 # this is in seconds
t_frames = dt_frame * np.arange(int(round(t_length_s / dt_frame)))

fig = plt.figure()
imgs = []
for t_frame in t_frames:
    t0 = t_frame
    t1 = t_frame + dt_frame
    m = (sim_t >= t0) & (sim_t < t1)

    frame_img = np.zeros((int(img_height/pool[0]), int(img_width/pool[1])))
    frame_img -= output_spikes_neg[m].sum(axis=0)
    frame_img += output_spikes_pos[m].sum(axis=0)
    frame_img = frame_img / np.abs(frame_img).max()

    img = plt.imshow(frame_img[:, ::-1], vmin=-1, vmax=1, animated=True)
    imgs.append([img])

ani = ArtistAnimation(fig, imgs, interval=50, blit=True)
HTML(ani.to_jshtml())

Discussion#

Notice the noise is HIGH. That’s because neurons are inherently noisy.

So again, why did we do this?

It seemed like a lot of work to just print the data again… right??

Well, now we know we can effectively send the data through neurons, which means we can now start processing the data with neurons and maintain the efficiencies we gained by collecting event-based data in the first place. COOL!

Contents