Research Interests

I am interested in animal behavior and how it is controlled by the brain. To study this, I develop tools for quantifying animal behavior from video data, for example pose estimation and action segmentation. I also develop tools for connecting that behavior to simultaneously recorded neural activity in order to better understand the neural information processing that underlies complex adaptive behavior.

Google Scholar

Lightning Pose

Lightning Pose (LP) is an industry-grade pose estimation software package that utilizes unlabeled frames to train better models. LP uses unsupervised training objectives that penalize the network whenever its predictions violate a variety of spatiotemporal constraints. It also uses a new network architecture that predicts the pose for a given frame using temporal context from surrounding unlabeled frames. We provide a browser-based app for data labeling and model training and evaluation.

paper | code | app

Semi-supervised behavioral segmentation

We introduce a semi-supervised approach to behavioral segmentation that leverages weak- and self-supervision. With this approach we show that a large number of unlabeled frames can improve supervised segmentation in the regime of sparse hand labels, and also show that a small number of hand labeled frames can increase the precision of unsupervised segmentation.

preprint | code

Partitioned Subspace VAE

The PS-VAE is a behavioral video analysis tool that combines the output of supervised pose estimation algorithms (e.g. DeepLabCut) with unsupervised dimensionality reduction methods (e.g. VAEs) to produce interpretable, low-dimensional representations of behavioral videos that extract more information than pose estimates alone. We show how these interpretable representations can be used to characterize the dynamics of behavior, as well as improve the precision of neural decoding analyses.

paper | code

BehaveNet

BehaveNet is a probabilistic framework that provides tools for compression, segmentation, generation, and decoding of behavioral videos. Compression is performed using convolutional autoencoders (CAEs). An autoregressive hidden Markov model segments the continuous CAE representation into discrete “behavioral syllables". Based on this generative model, we develop a novel Bayesian decoder that takes in neural activity and outputs probabilistic estimates of the behavioral video.

paper | code

Decoding in the presence of shared neural variability

We develop a new decoding framework for estimating stimulus identity from recorded neural population activity. Our framework exploits the low-dimensional structure of this activity, resulting in a linear estimator that is more efficient than those from other common linear decoding algorithms. Furthermore, this framework admits a straighforward nonlinear extension that compares favorably to other nonlinear decoding algorithms.

preprint

Characterizing shared variability in cortical neuron populations

Variability in neural population responses from early sensory areas often contains low-dimensional structure. Here we introduce two new classes of nonlinear latent variable models to characterize this structure. Both model classes rely on autoencoder neural networks for latent variable inference; one class models arbitrary nonlinear interactions while the other explicitly models additive and multiplicative modulations of stimulus responses.

paper | paper code | model code

A rectified latent variable model for analyzing neural population recordings

We propose the Rectified Latent Variable Model (RLVM) for analyzing neural population activity. The RLVM constrains latent variables to be both rectified and smooth. We demonstrate the advantages of these constraints using both simulated and experimental data, and show how initialization-dependent solutions can be improved by initializing model components with an autoencoder neural network.

paper | code