We report an electronic encoder (formed by a convolutional neural network) and a diffractive decoder (formed by spatially-structured diffractive layers) that are jointly optimized using deep learning to project super-resolved images at the output plane using a low-resolution spatial-light modulator (SLM). This diffractive super-resolution display performs ~4x optical super-resolution, corresponding to a ~16x increase in the space-bandwidth product. This diffractive display was experimentally demonstrated using 3D-printed diffractive decoders operating at the THz spectrum. Diffractive super-resolution image displays can be used to build compact, low-power, and computationally efficient HR projectors operating at visible wavelengths and other parts of the electromagnetic spectrum.

We show our recent progress on a Clements-type16x16 on-chip matrix processor based on silicon photonics and a new type of electro-optic digital-to-analog converters (EO DACs) with a higher signal-to-noise ratio. For the former, we developed a machine-learning-based calibration technique that involves theoretical modeling with circuit parameters (loss, phase error, splitting ratio, and crosstalks), which is adequate to obtain better fidelity for large-scale imperfect interferometers. After the calibration, we demonstrated a 16x16 identity matrix and several permutation matrices with a high signal-to-noise ratio and a well-known MNIST database classification task. For the latter, we developed low-loss and wavelength insensitive EO DACs consisting of 1:1 Y splitters and phase modulators that are useful for DAC-less input units for photonic accelerators.

Recent developments in the field of artificial intelligence have become a significant computational burden for current electronic hardware. In order to keep up with the increasing advancements in deep learning research, novel approaches to computation are required to address the slowing progress in computing performance and efficiency of electronic hardware. We will present our experimental results on a general-purpose optical processor that takes advantage of time multiplexing in integrated photonics and can perform not only dot products but also real-time correlation detection on stochastic bitstreams. This approach to optical processing has a significant compute efficiency advantage for long-bit sequences.

We present diffractive optical networks based on a series of passive light modulation surfaces, engineered/optimized using deep learning at the wavelength scale, to all-optically perform permutation operations, capable of achieving hundreds of thousands of interconnects between an input and an output field-of-view. We experimentally demonstrated, for the first time, a diffractive permutation network that operates at the THz part of the spectrum, realizing 625 interconnects between the input and output planes. These diffractive permutation networks can serve as channel routing and interconnection panels in the next-generation (6G) wireless communication systems with the carrier frequencies approaching THz-bands.

A deep learning network requires high-performance computer systems for solving complex problems with millions of parameters. In our lab, we develop a fully optical machine learning system that is based on the nonlinear four wave mixing process in multimode fibers. We exploit the optical nonlinear interactions between waves for developing a deep learning system faster than electronic based systems. finally, we resort to quantum light for realizing quantum deep learning system, which can bring the deep learning techniques to the quantum field. In this talk, we will present details of our novel system and discuss our preliminary results.

Computer technology is experiencing a shift as physical computing systems are becoming more sought after due to their ability to overcome limitations found in digital systems. The Ising machine is one such computing system with interest driven by its ability to solve combinatorial optimisation problems efficiently. Here we propose a novel Ising machine based on spontaneous symmetry-breaking in a Kerr optical resonator with an opto-electronic feedback mechanism. We will present numerical results demonstrating that our system may have inherent advantages over existing implementations due to the underlying symmetry-breaking mechanism.

We report a polarization-encoded diffractive network to perform multiple arbitrary complex-valued linear transforms within a single diffractive processor. An array of pre-selected linear polarizers is placed between the trainable isotropic diffractive layers, and distinct complex-valued linear transformations are individually assigned to different combinations of input/output polarization states. A polarization-encoded diffractive network performs the target linear transforms with negligible error when N ≥ P x I x O, where N is the number of trainable diffractive features/neurons, I and O denote the number of pixels at the input and output fields-of-view, respectively, and P represents the number of target linear transforms.

Reverse engineering the brain has been our dream and a grand challenge for many decades. The human brain has immense learning capabilities at extreme energy efficiencies and scale, which no artificial system has been able to match. We propose that utilization of a brain-derived—rather than a brain-inspired—architecture will lead to a paradigm shift, enabling the development of intelligent agents that can work in tandem with humans on complex tasks in noisy, unpredictable environments. Key to this paradigm shift will be the integration of hardware that emulates the processing characteristics of the brain with design principles based on fundamental aspects of neural plasticity and circuit design in the human brain. To achieve this grand challenge, we pursue a new neuromorphic computing paradigm enabled by modular integration of energy-efficient 3D-electronic-photonic-ICs (3D EPICs) with attojoule nanophotonic neurons in photonic neural networks.

Nonlinear Schrödinger Kernel is a new concept in computing using an optical system known as Nonlinear Schrödinger Kernel to perform machine learning acceleration. It connects information theory to nonlinear optical spectrum engineering, showing that this approach can effectively relieve the computational burden on the digital computer by elevating inference speed while reducing data dimension. A data encoding scheme is adopted to optimize the performance of the Nonlinear Schrödinger Kernel.

Data growth and AI demand low-latency hardware. ML is 90% convolutions (conv). (N-K), where N is input and kernel matrix size (assuming squared matrices). Classification ML kernel-weight changes are infrequent. Photonic ASICs benefit from compute-in-memory. Photonic memory accesses in 0.3ns and uses 100fJ/access, 100x less than SRAM. PCM photonic memory needed photon production and detection. C-band PIC detectors require 50nW above 30GHz. Assuming 1% laser wall-plug efficiency and 2dB per coupler, PCM-written P-RAM memory READ energy is 1fJ/access for OOK signal at 30GHz data rates, or 10fJ/access for greater bit resolution. Photonic connections 100x faster than SRAM. Off-chip learning kernels can execute conv-underlying MAC operations on electro-optic PICs. Specifically, utilizing fiber optical-based discrete components (non-PIC based) and PIC-based demonstrations show the possibility for efficient photonic MAC and hence conv. acceleration.

This study aimed to develop and implement a novel data encryption method that utilizes a hybrid processor Photonic Tensor Core and chaotic oscillators to generate an "infinite key" suitable for use with common encryption algorithms. To demonstrate its effectiveness, we built a prototype consisting of a hybrid processor simulator, chaotic oscillators, a key generator, an encryption/decryption tool, and a graphical user interface. We tested and inspected the tool using custom scripts and a graphical user interface, which allows two separate users to compare their respective results. In upcoming studies, we plan to expand the tool to accommodate multiple participants and develop a hardware prototype.

We report an all-optical object classification framework using a single-pixel diffractive network and spectrum encoding, classifying unknown objects through unknown random phase diffusers at the speed of light. Using this single-pixel diffractive network design, we numerically achieved a blind testing accuracy of 88.53%, classifying unknown handwritten digits through 80 unknown random diffusers that were never used during training. This framework presents a time- and energy-efficient all-optical solution for directly sensing through unknown random diffusers using a single pixel and will be of broad interest to various fields, such as security, biosensing and autonomous driving.

Reservoir computing is a powerful tool for creating digital twins of a target systems. They can both predict future values of a chaotic timeseries to a high accuracy and also reconstruct the general properties of a chaotic attractor. In this. We show that their ability to learn the dynamics of a complex system can be extended to systems with multiple co-existing attractors, here a four-dimensional extension of the well-known Lorenz chaotic system. Even parts of the phase space that were not in the training set can be explored with the help of a properly-trained reservoir computer. This includes entirely separate attractors, which we call "unseen". Training on a single noisy trajectory is sufficient. Because Reservoir Computers are substrate-agnostic, this allows the creation of conjugate autonomous reservoir computers for any target dynamical systems.

Here we introduce a Fourier-Theorem based convolution processors in silicon photonics. The systems leverages an algorithmic homomorphism that utilizes the Fourier transformation provided by a lens along with high-speed optoelectronic signal modulation and read-out. We demonstrate convolution filtering for image processing, convolutional neural network classification tasks. An on-chip lens performs the convolution operation, whereas electro-optic modulators perform the weighting in the Fourier domain at high-speed, followed by detection at a detector array after a 2nd Fourier lens, all on a PIC. Using this accelerator, we demonstrate image filtering and machine learning inference tasks. Given the high SWAP, these accelerators are useful for network-edge AI for the coming Industry-4.0 era.

Photonic neural networks are a highly promising computational system for AI-inspired future information processing. We have recently demonstrated the first fully implemented, photonic neural network realized in multimode semicondcutor lasers. The numerous laser modes acts as the systems neurons, which carrier diffusion and intra-cavity diffraction creating recurrent connections. I will discuss our recent result, where we push the realtime data-rate of the neural network towards GHz levels and use such systems to address highly relevant photonic-technology applications.

Utilizing the input-output transformation of ultrafast nonlinear pulse propagation in quadratic media, we experimentally construct a multilayer physical neural network to perform both audio and handwritten image classification. We introduce a hybrid in-situ in-silico backpropagation algorithm, physics-aware training, that is resilient to the simulation-reality gap, to train physical neural networks. The methodology for constructing and training physical neural networks applies to generic complex physical systems. To demonstrate its generality, we also built and trained physical neural networks out of analog electronic circuits and multimode mechanical oscillators to perform image classification.

A full start-to-end software (S2E) model of a laser system– including a mode-locked oscillator, chirped pulse amplification shaper, and nonlinear upconversion– can help expand high power laser system designs routinely tackled with human-centered methodologies. S2E models can even enable reverse engineering of a laser system, allow for more streamlined exploration of parameter spaces for experimental setups, or train machine learning models for optimization and tuning of these systems. We present a generalized S2E model targeted at generating data of the photoinjector laser system at SLAC’s LCLS-II for training neural networks for optimization and, eventually, active tuning of the photoinjector.

Recent work in deep learning has underscored the importance of measuring and understanding trends in model performance as a function of basic variables, such as the size of the training dataset, the number of model parameters, and the amount of compute. These trends often, though not always, are governed by power-law scaling. I will survey some of the existing empirical evidence for these so-called “scaling laws” and then discuss regimes where we have a theoretical understanding underlying these trends, based on joint work with collaborators. I will close by discussing connections to optical implementations of deep neural networks.

The novel field-resolved microscopy scheme of Quantum-probe Field Microscopy (QFIM) utilized fluorescence quanta to images local THz-electric waveforms [1]. In this contribution, we discuss the basis of the ultrafast microscopy scheme and the recovery of multi-Terahertz signals from fluorescence data. We elaborate fundamental aspects of time-domain sampling of electric waveforms and different strategies to recover response functions of systems under investigation. [1] M.B. Heindl, et al., “Ultrafast imaging of terahertz electric waveforms using quantum dots”, Light: Science & Applications 11, 2022.

We review the use of machine learning techniques in ultrafast dynamics in fiber-optics systems. We discuss how neural networks can be used to correlate the spectral and temporal characteristics of dissipative soliton lasers and predict nonlinear dynamics in optical fibers for a wide range of input conditions. We also show how machine learning algorithm allow for optimizing supercontinuum generation.

Linear optics has been long applied to image compression. However, it is widely known that nonlinear neural networks outperform linear models in terms of feature extraction and image compression. Here, we show a nonlinear multilayer optical neural network using a commercially available image intensifier as a scalable optical-to-optical nonlinear activation function. We experimentally demonstrated that nonlinear ONNs outperform linear optical linear encoders in a variety of non-trivial machine vision tasks at a high image compression ratio (up to 800:1). We have shown that nonlinear ONNs can directly process optical inputs from physical objects under natural illumination, which provides a new pathway towards high-volume, high-throughput, and low-latency machine vision processing.

An area of particular importance in developing advanced imaging techniques concerns 3D motion measurement in small-scale mechatronics and automated microscopy. One major drawback is related to complex motion measurement with 6 degrees of freedom. In the proposed work, the extraction of unknown metrics such as focusing distance, in plane and out-of-plane positioning from digital holograms is performed including real‐time constraints. This work explores extended computer micro-vision capabilities offered by combining digital holographic microscopy (DHM) and last generation of deep learning algorithms such as Vision Transformer (ViT) networks. Our experiments show that the reconstruction in-focus distance can be predicted in DHM with a high accuracy using tiny modified architectures of deep ViT networks and convolutional neural networks (CNN). We compare ViT and Tiny ViT models with deep CNN usually used in digital holography such as VGG16, LeNet and AlexNet.

The ability to tailor a specific electromagnetic field pattern along an arbitrary selected surface is interesting and of substantial importance considering its numerous immediate applications. It belongs to a class of inverse source problems, and as such it is challenging when only partial data is given. Here, a deep learning-based method that can map the electromagnetic field from an arbitrarily selected surface to a flat surface is presented. In addition, phase retrieval capability is demonstrated for finding both the phase and amplitude on an input flat surface from knowing only the amplitude on an arbitrarily selected surface.

The task of low-light image enhancement, in which the quality of images captured in low illumination conditions is enhanced, has high importance within industrial, security and scientific applications. In this paper we discuss Vision Enhancement via Virtual diffraction and coherent Detection (VEViD), a novel physics-inspired algorithm in which we map the physics of diffractive optics and phase detection to that of digital image processing. By applying virtual diffraction, we create a nontrivial spatial phase profile that encodes an enhanced version of the incoming image, a version in which the image’s previously unseen low-light details are made accessible to human and computer perception.

Deep neural network trained on physical losses are emerging as promising surrogates of nonlinear numerical solvers. These tools can predict solutions of Maxwell’s equations and compute gradients of output fields with respect to material properties in millisecond times which makes them very attractive for inverse design or inverse scattering applications. Here we demonstrate a neural network able to compute light scattering from inhomogeneous media in the presence of the optical Kerr effect from glass diffusers with a size comparable with the incident wavelength. The weights of the network are dynamically adjusted to take into account the intensity dependent refractive index of the material.

We report a computer-free imaging framework in which a set of transmissive diffractive layers were trained using deep learning to all-optically reconstruct arbitrary objects hidden by unknown, random phase diffusers. The image reconstruction of the object hidden behind a random and unknown phase diffuser is completed at the speed of light propagation through a thin, engineered diffractive volume. Our analyses provide a comprehensive guide for designing robust and generalizable diffractive imagers to all-optically see through random diffusers, which might be transformative for various fields, such as biomedical imaging, atmospheric physics, and autonomous driving.

In this talk, we will present PhyCV: the first physics-inspired computer vision library. PhyCV is a new class of computer vision algorithms directly derived from the equations of physics governing physical phenomena. The algorithms appearing in the first release emulate, in a metaphoric sense, the propagation of light through a physical medium with natural and engineered diffractive properties followed by coherent detection.

We present a diffractive camera that performs class-specific imaging of target objects, while all-optically and instantaneously erasing the objects from other classes during light propagation through thin diffractive layers, maximizing privacy preservation. We experimentally validated this class-specific camera design by 3D-printing the resulting diffractive layers (optimized through deep learning) and selectively imaging MNIST handwritten digits using the assembled camera system under terahertz radiation. The presented object class-specific camera is passive and does not require external computing power, providing a data-efficient solution to task-specific and privacy-aware modern imaging applications.

High-resolution images are necessary for various biological, clinical, and pathological applications. However, using objective lenses with high magnifications comes with its limitations: a smaller field of view, shorter working distance, limited depth of field, more intense illumination required, and a more complex illumination setup required. Single image super-resolution (SISR) could allow us to generate high-resolution images while still reaping the benefits of using low-magnification objectives. We developed SRFlow++, a conditional normalizing flow for SISR to empower our microscope with the ability to produce high-resolution (10x magnified) images of high fidelity and quality while using a low-magnification (4x) objective.