Significance: Our study introduces an application of deep learning to virtually generate fluorescence images to reduce the burdens of cost and time from considerable effort in sample preparation related to chemical fixation and staining.
Aim: The objective of our work was to determine how successfully deep learning methods perform on fluorescence prediction that depends on structural and/or a functional relationship between input labels and output labels.
Approach: We present a virtual-fluorescence-staining method based on deep neural networks (VirFluoNet) to transform co-registered images of cells into subcellular compartment-specific molecular fluorescence labels in the same field-of-view. An algorithm based on conditional generative adversarial networks was developed and trained on microscopy datasets from breast-cancer and bone-osteosarcoma cell lines: MDA-MB-231 and U2OS, respectively. Several established performance metrics—the mean absolute error (MAE), peak-signal-to-noise ratio (PSNR), and structural-similarity-index (SSIM)—as well as a novel performance metric, the tolerance level, were measured and compared for the same algorithm and input data.
Results: For the MDA-MB-231 cells, F-actin signal performed the fluorescent antibody staining of vinculin prediction better than phase-contrast as an input. For the U2OS cells, satisfactory metrics of performance were archieved in comparison with ground truth. MAE is <0.005, 0.017, 0.012; PSNR is >40 / 34 / 33 dB; and SSIM is >0.925 / 0.926 / 0.925 for 4′,6-diamidino-2-phenylindole/hoechst, endoplasmic reticulum, and mitochondria prediction, respectively, from channels of nucleoli and cytoplasmic RNA, Golgi plasma membrane, and F-actin.
Conclusions: These findings contribute to the understanding of the utility and limitations of deep learning image-regression to predict fluorescence microscopy datasets of biological cells. We infer that predicted image labels must have either a structural and/or a functional relationship to input labels. Furthermore, the approach introduced here holds promise for modeling the internal spatial relationships between organelles and biomolecules within living cells, leading to detection and quantification of alterations from a standard training dataset.
A proof-of-concept, compact, portable Fourier Ptychographic Microscope (FPM) to perform wide field-of-view, high spatial resolution imaging (<1 μm) for biosignature motility in liquid samples, is presented. The FPM has the potential method to be developed as a space-based payload for future landers destined to the Ocean Worlds. A portable FPM using an existing Fourier ptychography (FP) algorithm adapted for reconstruction is demonstrated. A NVIDIA Jetson Nano board and camera combined with FP, is used to computationally reconstruct sub-micron resolution images. Additionally, deep learning was employed to perform inferencing prediction which enables the on-edge FPM device.
In recent years, research efforts in the field of digital holography have expanded significantly, due to the ability to obtain high resolution intensity and phase images. The information contained in these images have become of great interest to the machine learning community, with applications spanning a wide portfolio of research areas including bioengineering. In this work, we seek to demonstrate a high fidelity simulation of holographic recording. By accurately representing diffraction, aberrations, and speckle introduced via propagation of a coherent light source through a series of optical elements and the object itself, we will accurately predict the optical interference of the object and reference wave at the recording plane. We will show that the optical transformation that predicts the complex field at the recording plane can be generalized for arbitrary holographic recording configurations using matrix optics. In addition, we will provide a detailed description of digital phase reconstruction and aberration compensation, for a variety of off-axis holographic configurations. Reconstruction errors will be presented for the various holographic recording geometries and complex field objects. The generalized holographic simulation described in this work will seek to motivate using the reconstruction of the simulated holograms to populate a database which can be used to train machine learning algorithms aimed at classifying relevant objects recorded through a variety of holographic setups.
Significance: We introduce an application of machine learning trained on optical phase features of epithelial and mesenchymal cells to grade cancer cells’ morphologies, relevant to evaluation of cancer phenotype in screening assays and clinical biopsies.
Aim: Our objective was to determine quantitative epithelial and mesenchymal qualities of breast cancer cells through an unbiased, generalizable, and linear score covering the range of observed morphologies.
Approach: Digital holographic microscopy was used to generate phase height maps of noncancerous epithelial (Gie-No3B11) and fibroblast (human gingival) cell lines, as well as MDA-MB-231 and MCF-7 breast cancer cell lines. Several machine learning algorithms were evaluated as binary classifiers of the noncancerous cells that graded the cancer cells by transfer learning.
Results: Epithelial and mesenchymal cells were classified with 96% to 100% accuracy. Breast cancer cells had scores in between the noncancer scores, indicating both epithelial and mesenchymal morphological qualities. The MCF-7 cells skewed toward epithelial scores, while MDA-MB-231 cells skewed toward mesenchymal scores. Linear support vector machines (SVMs) produced the most distinct score distributions for each cell line.
Conclusions: The proposed epithelial–mesenchymal score, derived from linear SVM learning, is a sensitive and quantitative approach for detecting epithelial and mesenchymal characteristics of unknown cells based on well-characterized cell lines. We establish a framework for rapid and accurate morphological evaluation of single cells and subtle phenotypic shifts in imaged cell populations.
We propose to use deep convolutional neural networks (DCNNs) to perform 2D and 3D computational imaging. Specifically, we investigate three different applications. We first try to solve the 3D inverse scattering problem based on learning a huge number of training target and speckle pairs. We also demonstrate a new DCNN architecture to perform Fourier ptychographic Microscopy (FPM) reconstruction, which achieves high-resolution phase recovery with considerably less data than standard FPM. Finally, we employ DCNN models that can predict focused 2D fluorescent microscopic images from blurred images captured at overfocused or underfocused planes.
We propose to use deep convolutional neural networks (DCNNs) to perform 2D and 3D imaging through scattering media. The inverse scattering problem is solved based on learning a huge number of training target and speckle pairs. The proposed technique does not rely on a reference beam, thus employs a simpler optical setup than previous techniques without the need to know the imaging model and optical processes. This lack of the need to know a prior model of the forward operator is very important since many optimization techniques are very sensitive to errors caused by the inaccuracy of the forward model.
Deep convolutional neural networks (DCNNs) offer a promising performance for many image processing areas, such as super-resolution, deconvolution, image classification, denoising, and segmentation, with outstanding results. Here, we develop for the first time, to our knowledge, a method to perform 3-D computational optical tomography using 3-D DCNN. A simulated 3-D phantom dataset was first constructed and converted to a dataset of phase objects imaged on a spatial light modulator. For each phase image in the dataset, the corresponding diffracted intensity image was experimentally recorded on a CCD. We then experimentally demonstrate the ability of the developed 3-D DCNN algorithm to solve the inverse problem by reconstructing the 3-D index of refraction distributions of test phantoms from the dataset from their corresponding diffraction patterns.
Digital holographic microscopy (DHM) provides label-free and real-time quantitative phase information relevant to the analysis of dynamic biological systems. A DHM based on telecentric configuration optically mitigates phase aberrations due to the microscope objective and linear high frequency fringes due to the reference beam thus minimizing digital aberration correction needed for distortion free 3D reconstruction. The purpose of this work is to quantitatively assess growth and migratory behavior of invasive cancer cells using a telecentric DHM system. Together, the height and lateral shape features of individual cells, determined from time-lapse series of phase reconstructions, should reveal aspects of cell migration, cell-matrix adhesion, and cell cycle phase transitions. To test this, MDA-MB-231 breast cancer cells were cultured on collagen-coated or un-coated glass, and 3D holograms were reconstructed over 2 hours. Cells on collagencoated glass had an average 14% larger spread area than cells on uncoated glass (n=18-22 cells/group). The spread area of cells on uncoated glass were 15-21% larger than cells seeded on collagen hydrogels (n=18-22 cells/group). Premitotic cell rounding was observed with average phase height increasing 57% over 10 minutes. Following cell division phase height decreased linearly (R2=0.94) to 58% of the original height pre-division. Phase objects consistent with lamellipodia were apparent from the reconstructions at the leading edge of migrating cells. These data demonstrate the ability to track quantitative phase parameters and relate them to cell morphology during cell migration and division on adherent substrates, using telecentric DHM. The technique enables future studies of cell-matrix interactions relevant to cancer.
Interferometric based techniques are often used for 3D quantitative phase imaging. While these techniques are sensitive to vibrations, non-interferometric intensity based techniques such as the transport of intensity equation (TIE) do not suffer from such a drawback. Phase reconstruction of phase objects using TIE technique is accomplished by recording several diffraction patterns at different observation planes through axially translating the CCD. In this paper, we purpose to use a spatial light modulator (SLM) in a modified 4f TIE optical setup to acquire 3D tomographic images of phase objects. This modified setup will reduce the acquisition time dramatically making the TIE technique useful for dynamic events such as biological samples. We illustrate how 3D phase objects can be reconstructed tomographically by constructing a rotating mechanism for the sample. At each angle of rotation, two diffraction patterns are captured by the CCD either sequentially or instantaneously with the help of a reference mirror. The reconstructed optical fields are tomographically recomposed to yield the final 3D shape using a tomographic backprojection technique. Finally, a reconfigurable hardware controlled by a GUI is employed to synchronize the CCD, the SLM and the rotating stage.
In this paper, we present detail analysis and a step-by-step implementation of an optimized fringe projection profilometry (FPP) based 3D shape measurement system. First, we propose a multi-frequency and multi-phase shifting sinusoidal fringe pattern reconstruction approach to increase accuracy and sensitivity of the system. Second, phase error compensation caused by the nonlinear transfer function of the projector and camera is performed through polynomial approximation. Third, phase unwrapping is performed using spatial and temporal techniques and the tradeoff between processing speed and high accuracy is discussed in details. Fourth, generalized camera and system calibration are developed for phase to real world coordinate transformation. The calibration coefficients are estimated accurately using a reference plane and several gauge blocks with precisely known heights and by employing a nonlinear least square fitting method. Fifth, a texture will be attached to the height profile by registering a 2D real photo to the 3D height map. The last step is to perform 3D image fusion and registration using an iterative closest point (ICP) algorithm for a full field of view reconstruction. The system is experimentally constructed using compact, portable, and low cost off-the-shelf components. A MATLAB® based GUI is developed to control and synchronize the whole system.
In this work we will extend the traditional TIE setup of phase retrieval of a phase object through axial translation of the CCD by employing a tunable lens (TL-TIE). This setup is also extended to a 360° tomographic 3D reconstruction through multiple illuminations from different angles by rotating the phase object. Finally, synchronization between the CCD, and the tunable lens is employed using a reconfigurable hardware to automate the 3D 360° tomographic reconstruction process.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.