1.

Introduction

In recent years, many research efforts have focused on wide field of view (WFOV) microscopic imaging methods with FOV significantly larger than that can be acquired directly by traditional microscopy.^1–8 The power of WFOV microscopic images has been appreciated in the applications of digital pathology and high-throughput screening with many commercially available products.^1,9,10 Currently, there are mainly four different techniques that are used to acquire WFOV microscopic images. The first technique, implemented in mainstream commercial products,¹ is to scan the sample under a traditional microscope scheme and stitch many small FOV images into a large FOV image. This technique requires only minimal modification of a traditional microscope, namely, adding only a two-dimensional translation stage. However, usually it can only achieve a moderate scanning speed and needs to solve the focusing problem during scanning. The second technique is based on digital holography and/or twin-image removal methods to acquire a wide-field image directly by image reconstruction.^2,3 This technique can be achieved with a compact and low-cost setup, and the FOV can be as large as the imaging sensor size. However, it is not suitable for efficient fluorescence imaging because of the difficulty to get an efficient reference beam in the fluorescence wavelength and also the incoherent nature of fluorescence light. The third technique is by shifting or tilting the illumination on the sample and reconstructing high-resolution images by acquiring multiple frames of low-resolution images.^4,5 The FOV of this technique will depend on the FOV of the low-resolution image acquired by either shadow imaging or with an imaging system and can be as large as the imaging sensor size. The imaging scheme for this technique is easy to implement and suitable for on-chip applications. However, this technique is relatively time-consuming compared with other techniques because many frames need to be acquired and processed to get the reconstructed image. The last technique is based on a multifocal scanning microscope, where a large-area focus grid is used to scan the sample and reconstruct the WFOV images accordingly.^6–8 This technique requires a scanning mechanism in contrast to the second and third techniques. However, it has the advantages of translation-stage-limited FOV and is most suitable to acquire fluorescence images⁸ among the four techniques, and it has a faster scanning speed compared to the first and third techniques.

Traditionally, scanning microscopes acquire images by Nyquist sampling with equally spaced sampling points. So WFOV microscopic images usually contain many more pixels compared to traditional microscopic images because of their large FOV, and a significantly large amount of data needs to be acquired during image acquisition. This will increase the burden for hardware design and memory usage and also results in limitations on acquisition speed. Fortunately, sparsity-based reconstruction techniques can possibly be used to loosen these requirements. In recent years, the sparsity-based compressive sampling or compressive sensing (CS) technique has attracted much research interests.¹¹ It has been proved that sparsity-based reconstruction from incomplete information is perfect if the sensing matrix and the image sparsity meet certain requirements.^12–14 The CS technique has been used in the field of optical imaging in the way of a single-pixel microscope.^15,16 And sub-Nyquist sparse sampling with a similar sparsity-based reconstruction method was also used to reconstruct optical coherence tomographic images consisting of axial line scans.^17,18

In our previous studies, we have implemented a WFOV multifocal scanning microscope using a regularly positioned focus grid and one-dimensional line scans that are equally spaced according to the Nyquist frequency sampling requirement.^6,7 In this paper, we used an irregular focus grid that was randomly positioned in one direction to scan the sample. In this case, the line scans were randomly spaced with average spacing larger than the Nyquist sampling requirement. Then the sparsity-based reconstruction technique used in CS reconstruction was applied to reconstruct the WFOV microscopic images. There are two distinct advantages of using sparse sampling in multifocal scanning microscopy. (1) The focus grid need not be regular and equally spaced, which will greatly loosen the requirement for fabrication of the optical elements used to generate the focus grid and thus reduce the fabrication cost. For example, the requirement for precise motorized stage movement in order to achieve equally spaced features for generating the focus grid will be loosened when using e-beam lithography or a focused ion beam, which is more challenging for wide FOV (several millimeters or centimeters) and usually incurs higher cost. Furthermore, the sparse nature of the focus grid can reduce the fabrication time significantly and thus reduce the cost. (2) Less data are required to reconstruct the WFOV sample image, which will considerably reduce the acquisition memory requirement as well as the acquisition time. For example, as we will show in later sections, it is possible to reconstruct reasonable images with 40% of missing data, which is significant for the memory requirement and acquisition time considering the amount of data that needs to be acquired for WFOV images, for example, $\sim 64\text{Mpixels}$ in our acquired image in Fig. 4(a). And obviously, the amount of data will be linearly scaled up for wider FOV images or when acquiring multiple images.

In the following sections, we will first describe our experimental setup and the sparsity-based reconstruction method for WFOV microscopic imaging, and then show the imaging results of the U.S. Air Force (USAF) target and onion skin cells while evaluating the performance of our reconstruction method with different percentages of missing data. Finally, we will discuss the prospects and limitations of our methods.

2.

Experimental Setup and the Sparsity-Based Reconstruction Method

The experimental setup of the WFOV multifocal scanning microscope is shown in Fig. 1(a). The Cartesian coordinate system is shown in Figs. 1(a) and 1(b). A diode-pumped solid-state laser source (MBL-III-473, Changchun New Industries Optoelectronics Technology Co., China) with a wavelength of 473 nm was guided through a single-mode fiber and collimated by an achromatic doublet lens with a focal length of 100 mm (GCL-010615, Daheng New Epoch Technology Inc., China), and then illuminated on a hologram with illumination power of 1.2 mW. The hologram was recorded on a silver-halide holographic plate using an in-line holography scheme with a semitransparent metal-coated mask, where an irregular aperture grid was patterned on the mask for generation of the irregular focus grid.^6,19 The irregular aperture grid with aperture size of $0.8\mu \mathrm{m}$ was made by a focused ion beam system (Nova 200, FEI Corp.). The collimated illumination beam was then converted by the hologram to an irregular focus grid with the same positions of the original irregular aperture grid on the mask. The transmission of the irregular focus grid through the sample was recorded by an imaging sensor (DMK22AUC03, The Imaging Sources, LLC) with a relay lens (#45-760, Edmund Optics) that imaged the focus grid onto the imaging sensor. The microscopic images were then acquired by scanning the sample in the $x$ direction with a translation stage actuated by a motor actuator (Z825B, Thorlabs).

Fig. 1

(a) Experimental setup of the WFOV multifocal scanning microscope, (b) part of the irregular focus grid on the imaging sensor, (c) positions of one row of spots in (b), and (d) typical full width at half maximum focal spot size measured by microscope with $60\times$ objective.

Figure 1(b) shows the irregular focus grid observed by the imaging sensor. As shown in the figure, the distance between adjacent focal spots in the same column was set as $30\mu \mathrm{m}$, and the separation of adjacent columns was also set as $30\mu \mathrm{m}$. However, the $y$ positions of focal spots in the same row were random with uniform distribution. Figure 1(c) shows the positions of one row of spots in Fig. 1(b), and it can be clearly seen that the row is not straight, but randomly curved instead, indicating random $y$ positions. During the experiment, the sample was scanned in the $x$ direction, and thus, the spacings between the scan lines were random and some spacings might be larger than the Nyquist sampling requirement, leading to missing line scans that we want to fill in by image reconstruction. In our experiment, we use 75 rows and different column numbers of the focus grid. Compared to the Nyquist sampling distance, set as $0.5\mu \mathrm{m}$ in the experiment, the ratio of missing line scans will depend on the column number that we use. The Nyquist sampling distance was determined according to spot sizes of the foci. Figure 1(d) shows a typical measurement of the full width at half maximum focal spot size by a microscope with a $60\times$ objective, which was $<1\mu \mathrm{m}$. In our experiment, the scanning speed was $80\mu \mathrm{m}/\mathrm{s}$ and the camera frame rate was $160\text{frames}/\mathrm{s}$, so the sampling distance was $0.5\mu \mathrm{m}$ in the $x$ direction, which was the Nyquist sampling distance suitable for a $1\mu \mathrm{m}$ focal spot size.

In the WFOV multifocal scanning microscope, the imaging resolution will be determined by the focal spot size. And the FOV will be determined by the scanning length in the $x$ direction and the extension of the focus grid in the $y$ direction, which are easily scalable.

To fill in the missing line scans in the image such that its pixel size is consistent with the Nyquist sampling requirement, we used the sparsity-based reconstruction method detailed as follows by observing that natural images are generally sparse in the wavelet domain.^11,13 In our experiment, the signal was detected in the spatial domain. The image we wanted to reconstruct can be described as a matrix $f$ of dimensions $n\times m$, where $n$ is the number of line scans that satisfies the Nyquist sampling requirement and $m$ is the number of pixels along the scanning direction, namely, the $x$ direction. In the experiment, what we obtained with our system was an image with sparsely spaced line scans, and the acquired data can be described as a matrix $y$ of dimension $p\times m$, where $p$ is the number of detected line scans and $p<n$. We can write $y=Rf$, where the matrix $R$ is of dimension $p\times n$. The elements of $R$ were set to be 1 or 0 according to the positions of detected line scans. The wavelet transform of $f$ can be written as $x=Sf$, where $S$ is the transform matrix. And we can also write $f=S^{-1}x$, where $S^{-1}$ denotes the inverse wavelet transform. Supposing the image is sparse in the wavelet domain, we can solve the following optimization problem to find the best estimation ($x^{\prime }$) of the image in the wavelet domain:

With the above method, we observed that the image reconstruction time was similar for a different percentage of missing data. We also notice that with the sparsity-based reconstruction method, the image reconstruction will be more time-consuming than direct reconstruction with Nyquist sampling, especially for large image size. So there is a trade-off between image acquisition time and reconstruction time for sparsity sampling and Nyquist sampling. This disadvantage of sparsity sampling could be ameliorated by using more powerful computation resources, for instance, better computer central processing units or high-speed graphics processing units.

3.

Experimental Results

We first use our WFOV multifocal scanning microscope to image the 1951 USAF target, as shown in Fig. 2. We scanned the USAF target along the $x$ direction for 4 mm and acquired 6000 frames by the imaging sensor in 37.5 s. We used 19, 29, and 50 columns along with 75 rows of the focus grid to acquire sparsely spaced images with 80, 60, and 40% missing line scans, where the total line scans was set to be 4689 according to the Nyquist sampling requirement of $0.5\mu \mathrm{m}$ sampling distance. The missing ratio did not linearly depend on the column number because some focal spots have the same $y$ position and scan the same line, and we kept the data from only one focal spot in this case.

Fig. 2

(a), (d), and (g) Sparsely spaced images of the center part of the USAF target with 80, 60, and 40% missing line scans, respectively. (b), (e), and (h) Reconstructed images of (a), (d), and (g), respectively, using the sparsity-based method. (c), (f), and (i) Enlarged images of the regions as indicated in (b), (e), and (h), respectively. (j) The WFOV image of the USAF target with size of $4.6\times 2.3\mathrm{mm}$, which was reconstructed from sparsely spaced image with 40% missing line scans. (k) Cross-section lines of the vertical and horizontal bars of group 8, element 1, with bar width of $1.95\mu \mathrm{m}$.

Figures 2(a), 2(d), and 2(g) show the sparsely spaced images of the center part of the USAF target ($512\times 512\text{pixels}$) with 80, 60, and 40% missing line scans, respectively, where the missing line scans are represented with black lines. Figures 2(b), 2(e), and 2(h) are the reconstructed images of Figs. 2(a), 2(d), and 2(g), respectively, using the sparsity-based method, and Figs. 2(c), 2(f), and 2(i) are enlarged images of the regions as indicated in Figs. 2(b), 2(e), and 2(h), respectively. It is obvious that the reconstruction quality is getting better as the missing data percentage drops. Figure 2(j) shows the WFOV image of the USAF target with size of $4.6\times 2.3\mathrm{mm}$ or $9315\times 4689\text{pixels}$ ($x\text{-}\text{direction}\times y\text{-}\text{direction}$), which were reconstructed from the sparsely spaced image with 40% missing line scans. Notice that the sawtooth shape in the left and right part of the image is caused by the different starting positions of line scans because of the different $x$ positions of the focal spots. As shown in the figure, the sparsity-based method can reconstruct part of the missing data caused by the sawtooth effect. Figure 2(k) shows the cross-section lines of the vertical and horizontal bars of group 8, element 1, with a bar width of $1.95\mu \mathrm{m}$ from Fig. 2(i) with 40% missing line scans. We notice that the vertical bars were better resolved compared to the horizontal bars. This is because the scan direction was along the $x$ direction, and the sampling distance was maintained as the Nyquist frequency with no missing data. And along the $y$ direction, the line scans were sparsely spaced and the missing line data were reconstructed using the sparsity-based method. With 40% missing line scans, the larger features can be reconstructed satisfactorily, but there are still residue errors for finer features. This should be caused by the high contrast of the USAF target, which leads to not totally fulfilling the sparsity requirement.

To further study the image reconstruction quality with a different ratio of missing line scans, we did simulations of an ideal USAF target with $512\times 512\text{pixels}$, as shown in Fig. 3(a). We randomly set the missing line scans of the USAF target with a different ratio of missing line scans and then did reconstruction with the sparsity-based method. We then calculated the relative error of the reconstructed image $f^{\prime }$ compared to the original image $f$ using the following equation:

Fig. 3

(a) Ideal USAF target for simulation to evaluate the performance of sparsity-based image reconstruction and (b) relative error against ratio of missing data in the sparsity-based image reconstruction.

Figure 3(b) shows the relative error against the ratio of missing data. We can see that the relative error will increase smoothly for small ratios and then sharply when the ratio is $>0.6$. And we can achieve reasonable relative error ($<0.1$) when the ratio is $<0.4$, that is, $<\sim 200$ missing line scans out of 512 total line scans. Notice that the relative error will depend on the sparsity of the image, and smaller errors are to be expected for images with more sparsity. Using the USAF target image in Fig. 3(a) as a reference, we calculated the relative errors for our acquired images shown in Figs. 2(c), 2(f), and 2(i), which are 0.28, 0.30, and 0.49 for the missing data ratios of 40, 60, and 80%, respectively. The errors for the missing data ratio of 40 and 60% are larger than the values shown in Fig. 3(b), probably because of image distortion caused by environment noise and imperfect movement of the scanning translational stage. The errors for a missing data ratio of 80% are roughly similar and significant, indicating that the reconstruction is unreliable, probably because the image might not be sparse enough in the wavelet domain. Nevertheless, we can still see acceptable details in the reconstructed images.

We next use our WFOV multifocal scanning microscope to image a microscope slide of onion skin cell, as shown in Fig. 4. We scanned the slide along the $x$ direction for 5 mm and acquired 10,000 frames by the imaging sensor in 62.5 s. We used 13, 30, and 52 columns along with 75 rows of the focus grid to acquire sparsely spaced images with 80, 60, and 40% missing line scans, where the total line scans was set to be 4811 according to the Nyquist sampling requirement of $0.5\mu \mathrm{m}$ sampling distance. Figure 4(a) shows the WFOV image of the onion skin cell with size of $6.5\times 2.4\mathrm{mm}$ or $\mathrm{13,278}\times 4811\text{pixels}$, which were reconstructed from the sparsely spaced image with 40% missing line scans. Figures 4(b), 4(c), and 4(d) show the sparsely spaced images of the region indicated in Fig. 4(a) ($512\times 512\text{pixels}$) with 80, 60, and 40% missing line scans, and Figs. 4(e), 4(f), and 4(g) show the corresponding reconstructed images. As expected, the reconstruction quality is better for smaller ratios of missing data, and we can clearly discern the cell nuclei and cell membranes of the onion skin cells.

Fig. 4

(a) WFOV image of microscope slides of onion skin cells with size of $6.5\times 2.4\mathrm{mm}$, which were reconstructed from sparsely spaced image with 40% missing line scans. (b), (c), and (d) Sparsely spaced images with 80, 60, and 40% missing line scans of the area indicated in (a). (e), (f), and (g) Reconstructed images of (b), (c), and (d), respectively, using the sparsity-based method.

4.

Discussion and Conclusion

In addition to the previously mentioned advantages of the sparse sampling version of the WFOV multifocal scanning microscope over that with Nyquist sampling, another important practical consideration is the adjustment of the focus grid orientation angle with respect to the sample scanning direction. In the Nyquist sampling version of our WFOV microscope,⁷ the angle must be finely adjusted such that the first focal spot in one row of focus grid scans the same line as the last focal spot of the next row, given a specific column number set by the Nyquist sampling distance. In the sparse sampling version of the WFOV microscope, however, the angle can be adjusted freely in a certain range, which will only change the percentage of missing line scans because of overlapping of scan lines by different focal spots. We just need to calibrate the angle once it is fixed. This advantage will greatly ease the experimental requirement and make the sparse sampling version of the WFOV microscope more practical.

We also notice that our current imaging system is sparsely sampled in only one dimension because of the one-dimensional scanning mechanism. It is also possible to implement two-dimensional (2-D) sparse sampling if a 2-D scanning mechanism is used. With the cost of a more complicated mechanical design and possibly slower acquisition time, the sparsity-based imaging reconstruction would be more effective because of a higher degree of randomness, which might decrease the coherence of the measurement matrix.²⁰

In summary, we proposed and implemented a WFOV multifocal scanning microscope with sparse sampling. To demonstrate its advantages of much less data acquisition compared to the Nyquist sampling requirement, we imaged the USAF target and an onion skin cell sample and showed the reconstruction results with 80, 60, and 40% missing line scans. The FOV of the onion skin cells sample image is $6.5\times 2.4\mathrm{mm}$, which is easily scalable with focus grid area and scanning length. Compared to the Nyquist sampling version, our imaging system allows for simpler experimental implementation, including easier fabrication of the hologram or diffractive optical elements to generate the focus grid, and also easier adjustment of the focus grid orientation angle with respect to the scanning directions.