Physics-enhanced machine learning for virtual fluorescence microscopy


This paper introduces a new method of data-driven microscope design for virtual fluorescence microscopy. Our results show that by including a model of illumination within the first layers of a deep convolutional neural network, it is possible to learn task-specific LED patterns that substantially improve the ability to infer fluorescence image information from unstained transmission microscopy images. We validated our method on two different experimental setups, with different magnifications and different sample types, to show a consistent improvement in performance as compared to conventional illumination methods. Additionally, to understand the importance of learned illumination on inference task, we varied the dynamic range of the fluorescent image targets (from one to seven bits), and showed that the margin of improvement for learned patterns increased with the information content of the target. This work demonstrates the power of programmable optical elements at enabling better machine learning algorithm performance and at providing physical insight into next generation of machine-controlled imaging systems.







Roarke Horstmeyer

Assistant Professor of Biomedical Engineering

Roarke Horstmeyer is an assistant professor within Duke's Biomedical Engineering Department. He develops microscopes, cameras and computer algorithms for a wide range of applications, from forming 3D reconstructions of organisms to detecting neural activity deep within tissue. His areas of interest include optics, signal processing, optimization and neuroscience. Most recently, Dr. Horstmeyer was a guest professor at the University of Erlangen in Germany and an Einstein postdoctoral fellow at Charitè Medical School in Berlin. Prior to his time in Germany, Dr. Horstmeyer earned a PhD from Caltech’s electrical engineering department in 2016, a master of science degree from the MIT Media Lab in 2011, and a bachelors degree in physics and Japanese from Duke University in 2006.

Unless otherwise indicated, scholarly articles published by Duke faculty members are made available here with a CC-BY-NC (Creative Commons Attribution Non-Commercial) license, as enabled by the Duke Open Access Policy. If you wish to use the materials in ways not already permitted under CC-BY-NC, please consult the copyright owner. Other materials are made available here through the author’s grant of a non-exclusive license to make their work openly accessible.