About Me
I am a machine learning research engineer with my Ph.D. from the University of Washington. I have experience in applying my deep learning expertise to problems in healthcare, bio-mechanics, and physics. I am passionate about applied AI and working closely with domain experts in developing AI solutions for complex science and engineering problems. I am currently looking for a position where I can apply my research and engineering skillset to complex and interesting problems.Machine Learning skills: computer vision, explainable AI, contrastive learning, transformer models, natural language processing (NLP) and large language models (LLMs), additive and multiplicative attention mechanisms, self-attention, multi-headed attention, video and 3D computer vision, DNN sparsification techniques (inc. pruning), model optimization and hyper-parameter tuning, model evaluation, custom DNN training and data pipelines.
Engineering skills: software development, collaborative use of Git, configuring and using containerized environments for GPU experiments (i.e., Docker), usable and modular code, code review.
Tools and frameworks: PyTorch, PyTorch Lightning, Weights and Biases, Tensorboard, TensorFlow, and JAX.
Research Projects
My academic research is focused on using deep learning to model complex dynamic systems. Some of the topics I researched in my thesis work are physical and biological systems modeling, network dynamics, sparse deep neural networks, and the relationship between task complexity and network structure.
MothPruning
We develop a framework that combines model predictive control on an established flight dynamics model and deep neural networks (DNNs) to create an efficient method for solving the inverse problem of flight control. We turn to natural systems for inspiration since they inherently demonstrate network pruning with the consequence of yielding more efficient networks for a specific set of tasks. This bio-inspired approach allows us to leverage network pruning to optimally sparsify a DNN architecture to perform flight tasks with as few neural connections as possible.
DNN Motif Analysis
We are quantifying the distribution of network motifs of sparse neural networks trained to model a biological control task. Complex networks across many domains are made up of statistically significant, subgraphs called network motifs. Network motifs have been shown to be indicative of network functionality in control systems. The DNNs in this study are trained to model insect flight control and are sparsified via neural network pruning. We have developed our own network motif mining algorithm based around using the masking matrices of the pruned networks.
pNIF
Model compression is important for modeling high-dimensional flow fields. NIF is a mesh-agnostic, dimensionality reduction paradigm for modeling complex spatio-temporal fields. NIF has a hypernet structure that isolates the spatial complexity from the parameter complexity and is capable of reconstructing complex spatio-temporal flow fields with parameters that amount to a fraction of the dimensionality of the dataset. This can be further reduced by neural network pruning. Here, we use the method of neural network pruning to further reduce the computational complexity of NIF.
Sparse ResNet-18
In this project, we sought to explore the relationship between image dataset complexity and the number of convolutional neural network (CNN) filters necessary for accurate image classification. We compared the maximum sparsity achieved via neural network pruning across networks trained on three different canonical datasets (MNIST, CIFAR-10, and CIFAR-100). We found that the maximum achievable sparsity is correlated with the task complexity of the dataset. Simpler tasks, such as classifying MNIST digits, can be done with fewer parameters than more challenging tasks (classifying 10 colored image classes or 100 colored image classes).
Sparse Sensor Placement
Sensor placement is an important and ubiquitous problem across the engineering and physical sciences for tasks such as reconstruction, forecasting and control. In this work, we develop two algorithms for optimizing sensor locations for use with shallow decoder networks (SDNs): one which is a linear selection algorithm based upon QR (Q-SDN), and one which is a nonlinear selection algorithm based upon neural network pruning (P-SDN). We demonstrate our sensor selection algorithms on two example data sets from fluid dynamics. Moreover, we provide a detailed comparison between our linear (Q-SDN) and nonlinear (P-SDN) algorithms with traditional linear embedding techniques (proper orthogonal decomposition) and QR greedy selection.
Shallow Recurrent Decoder Network
Sensing is a universal task in science and engineering. These tasks are exceptionally challenging to achieve with limited sensors, corrupt and missing data, and noisy measurements. Here, we propose a SHallow REcurrent Decoder (SHRED) neural network structure which incorporates a recurrent neural network (LSTM) to learn a latent representation of the temporal dynamics of the sensor, and a shallow decoder that learns an end-to-end mapping between this latent representation and the high-dimensional state space. SHRED enables faithful reconstructions with far fewer sensors, outperforms existing techniques when more measurements are available, and is agnostic towards sensor placement.
Publications
Zahn, Olivia, Jorge Bustamante Jr, Callin Switzer, Thomas L. Daniel, and J. Nathan Kutz. “Pruning deep neural networks generates a sparse, bio-inspired nonlinear controller for insect flight.” PLoS Computational Biology 18.9 (2022): e1010512. https://doi.org/10.1371/journal.pcbi.1010512
Zahn, Olivia, Thomas L. Daniel, and J. Nathan Kutz. "Motif distribution and function in sparsified deep neural networks." In progress. (exp. 2023).
Williams, Jan, Olivia Zahn, and J. Nathan Kutz. “Data-driven sensor placement with shallow decoder networks.” arXiv preprint arXiv:2202.05330 (2022).
Williams, Jan, Olivia Zahn, and J. Nathan Kutz. “Sensing with Shallow Recurrent Decoder Networks.” Nature Communications. Under review. arXiv preprint arXiv:2301.12011 (2023).