Week 14 Dynamics and Plasticity 14.1 Reservoir computing - Complex brain dynamics - Computing with rich dynamics Biological Modeling of Neural Networks: 14.2 Random Networks - stationary state - chaos

14.3 Stability optimized circuits Week 14 Dynamics and Plasticity Wulfram Gerstner EPFL, Lausanne, Switzerland - application to motor data 14.4. Synaptic plasticity - Hebbian - Reward-modulated 14.5. Helping Humans - oscillations

- deep brain stimulation Week 14-part 1: Review: The brain is complex Neuronal Dynamics Brain dynamics is complex 1mm 10 000 neurons 3 km wire motor cortex frontal

cortex to motor output Week 14-part 1: Review: The brain is complex Neuronal Dynamics Brain dynamics is complex -Complex internal dynamics -Memory -Response to inputs -Decision making -Non-stationary

-Movement planning -More than one activity value motor cortex frontal cortex to motor output Week 14-part 1: Reservoir computing

Liquid Computing/Reservoir Computing: exploit rich brain dynamics Maass et al. 2002, Jaeger and Haas, 2004 Review: Maass and Buonomano, Stream of sensory inputs Readout 1 Readout 2

Week 14-part 1: Reservoir computing 3 4 - if-condition on 1 -calculcate See Maass et al. 2007 Rich neuronal dynamics Week 14-part 1: Rich dynamics Experiments of

Churchland et al. 2010 Churchland et al. 2012 See also: Shenoy et al. 2011 Modeling Hennequin et al. 2014, See also: Maass et al. 2002, Sussillo and Abbott, 2009 Laje and Buonomano, 2012 Shenoy et al., 2011

Week 14-part 1: Rich neuronal dynamics: a wish list -Long transients -Reliable (non-chaotic) -Rich dynamics (non-trivial) -Compatible with neural data (excitation/inhibition) -Plausible plasticity rules Week 14 Dynamics and Plasticity 14.1 Reservoir computing - Complex brain dynamics - Computing with rich dynamics

Biological Modeling of Neural Networks: 14.2 Random Networks - rate model - stationary state and chaos 14.3 Hebbian Plasticity Week 14 Dynamics and Plasticity - excitatory synapses - inhibitory synapses

14.4. Reward-modulated plasticity Wulfram Gerstner EPFL, Lausanne, Switzerland - free solution 14.5. Helping Humans - oscillations - deep brain stimulation Week 14-part 2: Review: microscopic vs. macroscopic An (t )

I(t) Week 14-part 2: Review: Random coupling excitation Homogeneous network: - each neuron receives input from k neurons in network -each neuron receives the same (mean) external input

inhibition Week 14-part 2: Review: integrate-and-fire/stochastic spike arrival Stochastic spike arrival: excitation, total rate Re inhibition, total rate Ri Synaptic current pulses d u (u ueq) R dt

u u0 d u ( u u eq ) dt qe (t

f tk k, f qi (t k ', f '

EPSC RI ) f' tk' ) mean IPSC (t ) (t )

Firing times: Threshold crossing Langevin equation, Ornstein Uhlenbeck process Fokker-Planck equation Week 14-part 2: Dynamics in Rate Networks F-I curve of rate neuron d ri ri F ( w ij r j )

dt j Fixed point with F(0)=0 ri 0 Slope 1 Suppose: d 1 F '( 0 ) F (x 0) dx

stable unstable Suppose 1 dimension d x x F (w x) dt Exercise 1: Stability of fixed point d x x F (w x)

dt Fixed point with F(0)=0 x 0 Next lecture: 9h43 Suppose: d 1 F '( 0 ) F (x 0) dx

Suppose 1 dimension Calculate stability, take w as parameter d x x F (w x) dt Week 14-part 2: Dynamics in Rate Networks Blackboard: d ri ri F ( w ij r j ) Two dimensions!

dt j Fixed point with F(0)=0 ri 0 Suppose: d 1 F '( 0 ) F (x 0) dx Suppose 1 dimension

stable w<1 unstable w>1 d x x F (w x) dt Week 14-part 2: Dynamics in RANDOM Rate Networks d

ri ri F ( w ij r j ) dt j Fixed point: ri 0 stable R e( ) 1 unstable R e( ) 1

Random, 10 percent connectivity d 1 F '( 0 ) F (x 0) dx Chaotic dynamics: Sompolinksy et al. 1988 (and many others: Amari, ... Unstable dynamics and Chaos

d ri ri F ( w ij r j ) i ( t ) dt j chaos Rajan and Abbott, 2006 Image: Ostojic, Nat.Neurosci, 2014 Image: Hennequin et al. Neuron, 2014 Week 14-part 2: Dynamics in Random SPIKING

Networks d f u i u i w ij ( t t j ) dt j f Firing times: Threshold crossing Image: Ostojic, Nat.Neurosci, 2014 Week 14-part 2: Stationary activity: two different

regimes Stable rate fixed point, microscopic chaos Switching/bursts long autocorrelations: Rate chaos R e( ) 1 Ostojic, Nat.Neurosci, 2014

Week 14-part 2: Rich neuronal dynamics: a wish list -Long transients -Reliable (non-chaotic) -Rich dynamics (non-trivial) -Compatible with neural data (excitation/inhibition) -Plausible plasticity rules Week 14 Dynamics and Plasticity 14.1 Reservoir computing - Complex brain dynamics - Computing with rich dynamics

Biological Modeling of Neural Networks: 14.2 Random Networks - stationary state - chaos 14.3 Stability optimized circuits Week 14 Dynamics and Plasticity Wulfram Gerstner EPFL, Lausanne, Switzerland - application to motor data

14.4. Synaptic plasticity - Hebbian - Reward-modulated 14.5. Helping Humans - oscillations - deep brain stimulation Week 14-part 3: Plasticity-optimized circuit Re( ) 1

R e( ) 1 Image: Hennequin et al. Neuron, 2014 Optimal control of transient dynamics in balanced networks supports generation of complex movements Hennequin et al. 2014, Random stability-optimized circuit (SOC) Random

connectivity Week 14-part 3: Random Plasticity-optimized circuit Random Week 14-part 3: Random Plasticity-optimized circuit study duration of transients a1= slowest = most amplified

slope 1/linear theory slope 1/linear theory F-I curve of rate neuron Week 14-part 3: Random Plasticity-optimized circuit slope 1/linear theory a1= slowest = most amplified

F-I curve of rate neuron Optimal control of transient dynamics in balanced networks supports generation of complex movements Hennequin et al. NEURON 2014, Week 14-part 3: Application to motor cortex: data and model Churchland et al. 2010/2012

Hennequin et al. 2014 Quiz: experiments of Churchland et al. [ ] Before the monkey moves his arm, neurons in motor-related areas exhibit activity [ ] While the monkey moves his arm, different neurons in motorrelated area show the same activity patterns [ ] while the monkey moves his arm, he receives information which of the N targets he has to choose [ ] The temporal spike pattern of a given neuron is nearly the same, between one trial and the next (time scale of a few milliseconds)

[ ] The rate activity pattern of a given neuron is nearly the same, between one trial and the next (time scale of a hundred milliseconds) Week 14-part 3: Random Plasticity-optimized circuit Comparison: weak random Hennequin et al. 2014 Week 14-part 3: Stability optimized SPIKING network Classic sparse random connectivity (Brunel 2000) Random connections, fast

Stabilizy-optimized random connections distal connections, slow, (Branco&Hausser, 2011) structured Fast AMPA slow NMDA Overall: 20% connectivity 12000 excitatory LIF = 200 pools of 60 neurons 3000 inhibitory LIF = 200 pools of 15 neurons

Week 14-part 3: Stability optimized SPIKING network Spontaneous Classic sparse random connectivity (Brunel 2000) firing rate Neuron 1 Neuron 2 Neuron 3 Single neuron different initial conditions Hennequin et al. 2014

Week 14-part 3: Stability optimized SPIKING network Classic sparse random connectivity (Brunel 2000) Hennequin et al. 2014 Week 14-part 3: Rich neuronal dynamics: a result list -Long transients -Reliable (non-chaotic) -Rich dynamics (non-trivial) -Compatible with neural data (excitation/inhibition) -Plausible plasticity rules

Week 14 Dynamics and Plasticity 14.1 Reservoir computing - complex brain dynamics - Computing with rich dynamics Biological Modeling of Neural Networks: 14.2 Random Networks - stationary state - chaos

14.3 Stability optimized circuits Week 14 Dynamics and Plasticity Wulfram Gerstner EPFL, Lausanne, Switzerland - application to motor data 14.4. Synaptic plasticity - Hebbian - Reward-modulated 14.5. Helping Humans

- oscillations - deep brain stimulation Hebbian Learning = all inputs/all times are equal pre j post i w ij w ij F ( pre , post )

Week 14-part 4: STDP = spike-based Hebbian learning Pre-before post: potentiation of synapse Pre-after-post: depression of synapse Modulation of Learning = Hebb+ confirmation confirmation

Functional Postulate Useful for learning the important stuff w ij F ( p r e , p o s t , C O N F IR M ) local global Many models (and experiments) of synaptic plasticity do not take into account Neuromodulators. Except: e.g. Schultz et al. 1997, Wickens 2002, Izhikevich, 2007; Reymann+Frey 2007; Moncada 2007, Pawlak+Kerr 2008; Pawlak et al. 2010 Consolidation of Learning Success/reward

Confirmation -Novel -Interesting -Rewarding -Surprising Neuromodulators dopmaine/serotonin/Ach write now to long-term memory Crow (1968),

Fregnac et al (2010), Izhikevich (2007) Plasticity Stability-optimized curcuits - here: algorithmically tuned BUT - replace by inhibitory plasticity Vogels et al., Science 2011

avoids chaotic blow-up of network avoids blow-up of single neuron (detailed balance) yields stability optimized circuits Plasticity Readout - here: algorithmically tuned BUT Izhikevich, 2007 Fremaux et al. Success signal 2012

- replace by 3-factor plasticity rules Week 14-part 4: Plasticity modulated by reward Dopamine-emitting neurons: Schultz et al., 1997 Izhikevich, 2007 Fremaux et al. Success signal 2012 Dopamine encodes success= reward expected reward

Week 14-part 4: Plasticity modulated by reward Week 14-part 4: STDP = spike-based Hebbian learning Pre-before post: potentiation of synapse Week 14-part 4: Plasticity modulated by reward STDP with pre-before post: potentiation of synapse Quiz: Synpatic plasticity: 2-factor and 3-factor rules [ ] a Hebbian learning rule depends only on presynaptic and postsynaptic variables, pre and post

[ ] a Hebbian learning rule can be written abstractly as w ij F ( p r e , p o s t , C O N F IR M ) [ ] STDP is an example of a Hebbian learning rule [ ] a 3-factor learning rule can be written as w ij F ( p r e , p o s t , g lo b a lfa c to r ) [ ] a reward-modulated learning rule can be written abstractly as w ij F ( p r e , p o s t , s u c c e s s ) [ ] a reward-modulated learning rule can be written abstractly as

w ij F ( p r e , p o s t , n e u r o M O D U L A T O R ) Week 14-part 4: from spikes to movement How can the readouts encode movement? Week 14-part 5: Population vector coding Population vector coding of movements Schwartz et al.

1988 Week 14-part 4: Learning movement trajectories Population vector coding of movements 70000 synapses 1 trial =1 second Output to trajectories via population vector coding Single reward at the END of each trial based on similarity with a target trajectory Fremaux et al., J. Neurosci. 2010

Week 14-part 4: Learning movement trajectories QuickTime and a decompressor are needed to see this picture. Performance Fremaux et al. J. Neurosci. 2010 R-STDP

LTPonly Week 14-part 4: readout Plasticity can tune the network and Hebbian STDP - inhibitory connections, tuned by 2-factor STDP, for stabilization Vogels et al. 2011

Reward-modulated Success signal STDP for movement learning - Readout connections, tuned by 3-factor plasticity rule Fremaux et al. 2012 Last Lecture TODAY Exam: - written exam 17. 06. 2014 from 16:15-19:00

- miniprojects counts 1/3 towards final grade For written exam: -bring 1 page A5 of own notes/summary -HANDWRITTEN! Nearly the end: what can I improve for the students next year? Integrated exercises? Quizzes? Miniproject? Overall workload ?(4 credit course = 6hrs per week) Background/Prerequisites?

-Physics students -SV students -Math students Slides? videos? Week 14 Dynamics and Plasticity 14.1 Reservoir computing - Review:Random Networks - Computing with rich dynamics

Biological Modeling of Neural Networks: 14.2 Random Networks - stationary state - chaos 14.3 Stability optimized circuits Week 14 Dynamics and Plasticity Wulfram Gerstner EPFL, Lausanne, Switzerland

- application to motor data 14.4. Synaptic plasticity - Hebbian - Reward-modulated 14.5. Helping Humans - oscillations - deep brain stimulation Week 14-part 5: Oscillations in the brain Individual neurons

fire regularly two groups alternate Week 14-part 5: Oscillations in the brain Individual neurons fire irregularly Week 14-part 5: STDP Week 14-part 5: STDP during oscillations Oscillation near-synchronous spike arrival

Week 14-part 5: Helping Humans big weights Pfister and Tass, 2010 small weights Week 14-part 5: Helping Humans: Deep brain stimulation Parkinsons disease: Symptom: Tremor -periodic shaking - 3-6 Hz

Brain activity: - thalamus, basal ganglia - synchronized 3-6Hz in Parkinson - Deep brain stimulation Benabid et al, 1991, 2009 Week 14-part 5: Helping Humans Abstract Dynamical Systems

View of Brain states healthy pathological Week 14-part 5: Helping Humans: coordinated reset Pathological, synchronous Coordinated reset stimulus Week 14-part 5: Helping Humans Tass et al. 2012

Theoretical concepts can help to understand brain dynamics contribute to understanding plasticity and learning inspire medical approaches can help humans The End Plasticity and Neuronal Dynamics now QUESTION SESSION! Questions to Assistants possible until June 1

The end and good luck for the exam! Week 14-part 3: Correlations in Plasticity-optimized circuit Hennequin et al. 2014,