Towards VLSI Spiking Neuron Assemblies as - ETH E

Diss. ETH No. 19456
Towards VLSI Spiking Neuron Assemblies
as General-Purpose Processors
A dissertation submitted to the
ETH ZURICH
for the degree of
Doctor of Sciences
presented by
EMRE ¨ OZG ¨ UR NEFTCI
Ing. phys. dipl. EPFL
born 8 July 1982
citizen of Geneva, Switzerland
accepted on the recommendation of
Prof. Dr. Rodney J. Douglas, examiner
Dr. Giacomo Indiveri, co-examiner
Prof. Dr. Stefano Fusi, co-examiner
Dr. Elisabetta Chicca, co-examiner
2010

Abstract
For the last two decades, the field of Neuromorphic Engineering has attempted to emulate
the principles observed in neural systems in Very Large Scale Integration (VLSI) technology.
One endeavor of Neuromorphic Engineering is to mimic biological neural information processing
systems by implementing electronic analogs of spiking neurons as computational primitives.
Neuromorphic Engineering plays an important role in testing and validating hypotheses from the
neuroscience community while developing novel computational architectures for solving practical
problems in real-time.
Although Neuromorphic Engineering is clearly progressing from a technical point of view, practi-
cal applications are hindered by two unsolved problems: 1) the technical difficulty of configuring
the parameters of the silicon neurons to match a given theoretical model and 2) the lack of
general-purpose computational models that can be mapped onto spiking neural networks.
This thesis provides solutions to these two problems using a special neural circuit inspired by
the cortex: the Soft Winner–Take–All (sWTA). Similar to how transistors and logic gates have
become the elementary computational units of general-purpose digital processors, neuromorphic
spiking multi-neuron chips implementing sWTA have been proposed as the elementary units
which could be composed into a general-purpose neuromorphic processor. The goal of this thesis
is to define the first configuration language which can map a high-level computational model
onto VLSI neuromorphic spiking neurons, starting from their low-level parameters, and thus
raise the possibility of using multi-neuron neuromorphic chips as general-purpose processors.
To achieve this goal, I first deal with the technical difficulty of configuring the parameters of
the silicon neurons. In particular, I describe a model-based parameter translation technique
for systematically mapping the parameters of theoretical models of spiking neurons onto the
bias voltages and currents used to configure the multi-neuron chips. In addition, using Dy-
namic Parameter Estimation (DPE), a novel technique that utilizes synchronization as a tool
for dynamically coupling experimentally measured data to its corresponding model, I can use
transients in neural activity to efficiently estimate all the parameters of spiking neural networks.
These two techniques allow me to systematically measure and set the key properties of a VLSI
sWTA network.
However, configuring the properties of the individual VLSI sWTA network in a multi-sWTA
system is not sufficient to guarantee global stability. For determining boundary conditions for
which large neural systems are guaranteed to remain stable, I combine the use of contraction
theory with the Linear Threshold Unit (LTU) formalism for modeling spiking neurons. In con-
traction theory, these boundary conditions can be expressed solely in terms of the key properties
v

vi Abstract
of the sWTA networks which compose the system and the relations between them. As an ex-
ample, I use networks of sWTA to implement a factor graph-like model in which the recurrent
interactions are reminiscent of belief propagation algorithms applied to graphs. Using simple
relations between several sWTA networks, I developed a multi-chip system that computes basic
linear and non-linear relations between the variables represented by sWTA networks.
The crucial step towards the goal of this thesis is the ability to program the neuromorphic chip
according to a known computational model. An interesting model is the state machine, because
it can model behavior conditioned on previously observed cues, which is the hallmark of intel-
ligent systems. Neuroscientists argue that the neural analog of state machines is supported by
working memory. They have demonstrated that recurrently coupled neural networks can main-
tain persistent activity states and support the conditional state switchings characterizing the
transitions in state machines. Using the parameter configuration technique, I devised a method
which configures a multi-chip VLSI system composed of three sWTA networks to implement
any Finite State Machine (FSM). I demonstrate a fully functional state machine and use these
principles to process sensory data from a silicon retina.

R´esum´e
Depuis les deux derni`eres d´ec´ennies, le domaine de l’ing´enieurie neuromorphique s’´efforce de
simuler les principes des syst`emes nerveux dans la technologie VLSI. Clairement, le domaine
progresse du point de vue technique, toutefois, les applications pratiques continuent d’ˆetre
empˆech´ees par deux probl`emes irr´esolus: 1) la difficult´e technique consistant `a configurer les
param`etres des neurones VLSI afin qu’ils correspondent `a ceux du mod`ele th´eorique. 2) l’absence
d’un mod`ele de calcul `a usage g´en´eral qui peut ˆetre transf´er´e dans les r´eseaux de neurones.
Cette th`ese fournit les solutions `a ces deux probl`emes en utilisant un circuit neuronal VLSI in-
spir´e du cortex: le “soft Winner-Take-All” (sWTA). De la mˆeme mani`ere dont les transistors et
les fonctions logiques sont devenus les unit´es de calcul ´el´ementaires des processeurs d’aujourd’hui
(tels que ceux dans nos ordinateurs), les puces neuromorphiques multi-neurones impl´ementant le
sWTA sont propos´ees en tant qu’unit´es ´el´ementaires pouvant composer un processeur neuromor-
phique `a usage g´en´eral. Le but de cette th`ese est de d´efinir une proc´edure de synth`ese neuronale
capable de syst´ematiquement configurer les VLSI sWTAs, en commen¸cant par les param`etres de
bas-niveau des neurones VLSI qui le composent et en proc´edant vers le haut-niveau, de mani`ere
`a fournir le calcul d´esir´e.
Pour atteindre ce but, je traite tout d’abord la difficult´e technique consistant `a configurer
les param`etres des neurones VLSI. En particulier, je d´ecris une technique de traduction de
param`etres bas´ee sur les mod`eles des neurones implement´es dans le VLSI. Cette technique per-
met d’appliquer les voltages n´ecessaires pour configurer les param`etres des puces multi-neurones.
De plus, en utilisant le “Dynamical Parameter Estimation” (estimation dynamique des param`etres)
(DPE), une nouvelle technique qui exploite la synchronisation des syst`emes dynamiques pour
coupler les measures exp´erimentales `a un mod`ele math´ematique qui y correspond, je peux utiliser
les ´etats transitoires de l’activit´e neuronale pour estimer simultan´ement tous les param`etres du
r´eseau de neurones.
Ces deux techniques me permettent de syst´ematiquement mesurer et de contrˆoler les param`etres
cl´es du sWTA.
Cependent, configurer les propri´et´es de chaque sWTA dans un syst`eme compos´e d’un grand
nombre de sWTA n’est pas suffisant pour garantir une stabilit´e globale. Pour d´eterminer les
conditions de bord garantissant la stabilit´e des grands syst`emes, je combine la th´eorie de la
contraction avec le formalisme des Linear Threshold Units qui approximent l’activit´e des spiking
neurones. Dans la th´eorie de la contraction, ces conditions de bords peuvent ˆetres exprim´ees
uniquement en termes des propri´et´es cl´es des sWTA qui composent le syst`eme et les relations
entre eux.
vii

viii R´esum´e
J’illustre cette th´eorie dans l’exemple d’un syst`eme impl´ementant un mod`ele semblable `a un
graphe factoriel, dans lequel les connections bidirectionnelles jouent le rˆole des algorithmes
de ”belief propagation” appliqu´ees `a de tels graphes. En utilisant des relations simples entre
plusieurs sWTA, j’ai d´evelopp´e un syst`eme multi-puces qui calcule les relations lin´eaires et non-
lin´eaires entre les variables repr´esent´ees par les sWTA.
L’´etape cruciale vers le but de cette th`ese r´eside dans la capacit´e de programmer les circuits neu-
romorphiques pour qu’ils simulent un mod`ele de calcul. L’automate fini est un mod`ele int´eressant
parce qu’il peut simuler des comportements d´ependents des signaux per¸cus ant´erieurement, ce
qui est un trait caract´eristique des syst`emes intelligents. Les neuroscientifiques soutiennent
que l’´equivalent neuronal de l’automate fini est soutenu par une m´emoire `a court terme. Ils
d´emontrent que des mod`eles de r´eseaux coupl´es de mani`ere bidirectionnelle sont capables de
maintenir les ´etats persistants et supporter les transitions entre ces ´etats. En utilisant la tech-
nique de la traduction des param`etres, j’ai concu une m´ethode pour automatiquement config-
urer trois sWTAs de mani`ere `a impl´ementer un automate fini d´esir´e. Je d´emontre un auto-
mate fini enti`erement fonctionnel que j’utilise pour traiter l’information fournie par une r´etine
´electronique.