3.1.

Light Collection

First, we determine the light collection efficiency of a single-aperture system. Consider a setup with a scene emitting the radiance $L$, a lens with diameter $D$ and effective focal length $f$, and an image sensor [Fig. 3(a)]. The sensor has the extent $w\times h$, divided into $n_{\mathrm{x}}\times n_{\mathrm{y}}$ pixels with a pitch of $p_{\mathrm{x}}$. From the image plane, the lens subtends a solid angle of

Fig. 3

Geometric properties of a single-aperture system (a) and two multi-aperture systems, supersampling (b) and segmenting (c) with lens diameter $D$, focal length $f$, image height $h$, and field of view $\alpha$. The lens subtends an angle $\mathrm{\Omega }$ at the image plane.

As the aperture takes on the radiance of the scene,¹² the sensor receives an irradiance of

We discuss a supersampling multi-aperture system next [Fig. 3(b)]. To decouple sampling rate from pixel pitch, the single optical system is replaced by $N\times N$ channels side by side, with the supersampling factor $N$. In case of the eCLEY, the supersampling factor $N$ is 2, though the number of channels is higher because the FOV is segmented.

Each channel is a scaled version of the original system (see Ref. 2), so $f^{\prime }=f/N$, $D^{\prime }=D/N$ and each system retains the F-number of the original camera. Therefore, in each system, ${\mathrm{\Omega }}^{\prime }=\mathrm{\Omega }$. Consequently, each pixel records the same flux ${\mathrm{\Phi }}_{\mathrm{pix}}$ as in the single-aperture case. As the system samples the same FOV (solid angle) as the original system with the same sampling rate, the total amount of samples—or pixels—stays the same. Therefore,

Next, we segment the FOV $\alpha$ of the camera into $M\times M$ channels [Fig. 3(c)]. In case of the eCLEY, $M$ is 8 horizontally and 6 vertically. The geometry of each of these channels is identical to that of the original optical system: Both $f$ and $D$ stay the same. The FOV of each channel is limited, however, by reducing the image size in each channel. The viewing direction of the channel is selected by introducing a lateral offset between optical system and image. Again neglecting distortion and vignetting, in each channel, partial FOV is $\alpha /M$ and image size is $w/M\times h/M$. As sampling rate and pixel size are kept the same, each channel now uses $n_{\mathrm{x}}/M\times n_{\mathrm{y}}/M$ pixels. If either distortion or vignetting are not corrected in the optical system, they affect single-aperture and multi-aperture systems in the same way.

Because the focal length is still $f$ and the aperture diameter is still $D$, $\mathrm{\Omega }$ remains the same and ${\mathrm{\Phi }}_{\mathrm{pix}}^{\prime }={\mathrm{\Phi }}_{\mathrm{pix}}$. As the total amount of pixels in the system does not change, ${\mathrm{\Phi }}_{\mathrm{tot}}^{\prime }={\mathrm{\Phi }}_{\mathrm{tot}}$.

In summary, both segmenting and supersampling multi-aperture systems collect the same amount of light as single-aperture systems, as long as the $F$-number and the total photodetector area are kept constant.

3.2.

Sharpness

In this section, we will first examine the effects of supersampling on image sharpness. Segmentation of the FOV will be of relevance in the course of the discussion.

By using supersampling, a digital camera can be made thinner without sacrificing sampling rate in object space and without requiring a smaller pixel pitch. To retain actual optical resolution in object space along with sampling rate, however, the MTF of the channels in image space has to keep up with the sampling rate.

Supersampling multiplies the image plane sampling frequency $f_S$ and the Nyquist frequency $f_{\mathrm{Ny}}$ by a factor of $N$. Therefore, the MTF should now show significant modulation up to $f_{\mathrm{Ny}}^{\prime }=N·f_{\mathrm{Ny}}$. Consequently, it has to improve considerably.

The MTF of a camera is the product of the MTF of the lens and the sensor, where the sensor MTF consists of a geometrical component and a component resulting from crosstalk between pixels:

${\mathrm{MTF}}_{\mathrm{G}}$ describes spatial integration over the photodetector. For square photosites, it is the Fourier transform of the rect function with the width of the photosensitive area $p_{\mathrm{p}}$:

The pixel pitch $p_{\mathrm{x}}$ stays the same. While we are still free to choose a smaller $p_{\mathrm{p}}$, light sensitivity decreases with photosensitive area, or $p_{\mathrm{p}}^2$. Therefore, we assume ${\mathrm{MTF}}_{\mathrm{G}}$ to be constant.

Crosstalk depends on the chief ray angle of light incident on the sensor and on sensor technology. Neither of them changes for multi-aperture systems. Therefore, ${\mathrm{MTF}}_C$ is constant as well.

The burden of improving the system MTF is therefore placed entirely on the optical MTF. As described by Lohmann,¹³ if an optical system is scaled by the factor $1/N$, the area of an image point $A_p$ scales as

When diffraction is neglegible, the diameter of an image point is

Segmenting the FOV also has beneficial effects. Many of the Seidel aberrations depend on field height $h$.¹² Field curvature and astigmatism, for example, increase with $h^2$. Therefore, segmenting the FOV into $M$ parts reduces aberrations accordingly.

However, quantifying the benefit exactly is not possible so easily. The well-known scaling laws for Seidel aberrations only apply to imaging with a single lens. In practice, aberrations are partially corrected with multilens systems, whose behaviour is more complex. This is true even for low-cost mass-market cameras for mobile devices. With a certain amount of correction, higher-order aberrations cannot be neglected any more; these aberrations also defy description by simple scaling laws.

In conclusion, optical MTF is indeed improved considerably by scaling. This is necessary to retain optical resolution in object space. As an example on how this works out, Fig. 4 shows the MTF of a system with aberrations ($N=1$) and the effect of scaling down this system ($N>1$). First, only optical MTF is plotted on an absolute frequency axis (a). Optical MTF is improved as expected for increasing $N$. However, when the $f$ axis is normalized to the sampling frequency $f_S$, which scales with $N$, improvement is less apparent (b). When we include pixel MTF, system MTF is similar for all $N$ (c). Therefore, object-space sharpness of the supersampled systems is comparable to the original system. Increasing $N$ further still improves optical MTF, but pixel MTF cancels this gain.

Fig. 4

Scaling effects for a system with two optical surfaces and $f=4\mathrm{mm}$, F2.4, simulated on-axis MTF curves. (a) Optical MTF improves for scaling the optical system down. Supersampling factors $N=1$ to 3; $N=1$ is the original system. Plotted on absolute frequency axis (cycles/mm). (b) The same curves plotted on a frequency axis relative to the image-space sampling frequency $f_S$, which scales with $N$. (c) Optical MTF multiplied with geometrical pixel MTF. (d) Improvement in optical MTF is limited by diffraction effects, shown by scaling the system down by a factor of 4.

Enhancement to the optical MTF itself is limited by diffraction, which is independent of system scaling. This is illustrated in Fig. 4(d). Here, we used the same optical system as in Fig. 4(a), but scaled it down by 4, so $f$ is now 1 mm. Again, optical MTF is improved for $N=2$, but improvement is limited by diffraction (dashed line). The resulting object space sharpness for $N=2$ is lower than the sharpness of the original system.

3.3.

Manufacturing Tolerances

When manufacturing a lens system, deviations in lens curvatures, distances, decenter and tilt degrade the system performance. When scaling a lens system down, deviations have to be smaller as well, or performance is compromised. As a simple example, consider a single thin lens with focal length $f$ and diameter $D$ positioned so that it focuses light from point $P$ onto an image plane (Fig. 5). When the lens is moved from its correct image plane distance $f$ by a deviation $\mathrm{\Delta }s$, defocus leads to an image point diameter $d_P=\mathrm{\Delta }s·D/f$. The smaller the system, the smaller $d_P$ has to be to retain sharpness. Accordingly, $\mathrm{\Delta }s$ has to be smaller as well.

Fig. 5

Image point diameter $d_P$ of a defocused optical system with focal length $f$ and lens diameter $D$, with the image plane moved by $\mathrm{\Delta }s$.

The same is true for the focal length of the lens: For a plano-convex lens, $f$ is proportional to lens radius $R$,¹² so a deviation $\mathrm{\Delta }R$ leads to a new focal length $f^{\prime }$ with $\mathrm{\Delta }f=f-f^{\prime }$. $\mathrm{\Delta }f$ effectively is a defocus shift $\mathrm{\Delta }s$, leading to an image point diameter analogous to a lens shift.

For perspective, with current pixel technology, $d_P<2\mu \mathrm{m}$ is desirable. This requires a focus shift of less than $2d_P=4\mu \mathrm{m}$.

The ability to meet the required tolerances depends on the technology that is used to manufacture and assemble the lens components. A suitable technology for multi-aperture systems is wafer-level optics (WLO), as multiple lenses side by side are manufactured and aligned in parallel. Assembly of the lens components can be achieved with the required micron precision.¹⁴

Critical, however, is precision during replication of the lens components. Lenses are manufactured from certain polymers by molding and ultraviolet curing. During hardening, these materials shrink significantly. The amount of shrink is proportional to the lens volume. Lens volume grows with the square of the lens radius and linearly with lens sag. Therefore, small lenses with low sags are preferable. Molding tools are adjusted to anticipate shrink; however, shrink has a certain spread that is proportional to shrink itself. The hardened lenses therefore still have form deviations that scale with $N^3$.

Using a multi-aperture architecture—either supersampling or segmenting—decreases lens diameters. For a supersampling factor of $N$, lens diameter decreases by $N$, lens sag also decreases by $N$ and we can expect deviations to decrease with the cube of $N$. Segmenting the FOV also reduces lens diameters, having a similar effect on deviations.

In conclusion, while tolerances have to be tighter for scaled-down lens systems, the fact that small lenses can be manufactured with less shrink makes it easier to meet these tolerances. Therefore, sharpness of actual, mass-manufactured camera systems can benefit significantly from a multi-aperture architecture. This result contradicts the theoretical analysis in Sec. 3.2, which suggested that multi-aperture systems can at best reach a performance comparable to single-aperture systems.

3.4.

Volume

We already established that system thickness is reduced by supersampling. In some applications, however, total system volume is more relevant than thickness. Therefore, we now examine how multi-aperture system volume $V^{\prime }$ compares to that of a single-aperture system $V$. We again treat the two different architectures (supersampling and segmented FOV) separately. In both cases, we first derive the footprint of the system. It is given by either sensor footprint $A_{\mathrm{sens}}$ or total aperture area $A_{\mathrm{tot}}$, depending on which one is larger. In the single-aperture case, $A_{\mathrm{tot}}$ is simply the area of the single system aperture. The values for the multi-aperture system are $A_{\mathrm{sens}}^{\prime }$ and $A_{\mathrm{tot}}^{\prime }$, which is now the sum of all individual aperture areas $A^{\prime }$. Next, we derive system height. In both cases, system height scales with effective focal length $f$. To $f$, a part of the optical system thickness $h_{\mathrm{opt}}$ is added, depending on system complexity and placement of the principal planes. We disregard the thickness of the image sensor, sensor carrier and casing, as these values are small compared to the focal length and are not affected by the system architecture.

Supersampling: As noted in Sec. 3.1, neither the pixel pitch nor the total number of pixels on the sensor change. Therefore, $A_{\mathrm{sens}}^{\prime }=A_{\mathrm{sens}}$. This is also true for $A_{\mathrm{tot}}^{\prime }$:

Segmentation of FOV: Again, pixel size and number stay the same, so $A_{\mathrm{sens}}^{\prime }=A_{\mathrm{sens}}$. However, the single-aperture with area $A_{\mathrm{tot}}$ is now replaced with $M$ copies of the original aperture. Aperture area therefore is increased:

The proportion of $A_{\mathrm{tot}}$ to $A_{\mathrm{sens}}$ in a camera is approximately the proportion of the corresponding lengths:

Miniaturized cameras tend to have a large FOV. If we assume $N=2.8$ and $\alpha =70°$, $D_{\mathrm{sens}}/D\approx 4$. Therefore, the sensor width is larger than the lens diameter and system footprint is given by $A_{\mathrm{sens}}$ for $M\le 4$.

Effective focal length is not affected. Reduced optical system complexity in each channel, however, decreases $h_{\mathrm{opt}}^{\prime }$ slightly. Therefore, system volume $V^{\prime }$ is smaller than $V$ for moderate segmentation of FOV, but increases with $M^2$ for large $M$.

This analysis does not consider additional volume consumed by the system casing, structures for suppressing stray light or walls separating channels. The latter are needed to prevent crosstalk between channels. In current systems such as the eCLEY, structures for crosstalk suppression do consume a considerable amount of space between channels. They therefore increase the total volume of the system and lead to unused areas on the image sensor. For reducing this waste of space and sensor area, very thin vertical or slanted walls have to be manufactured. Techniques for cheaply fabricating these structures are currently being developed.

4.1.

Algorithm

We treat each recorded pixel as a measurement of the light incident on the camera from a specific direction. The pixel viewing directions are derived from the model of the optical system; it includes effects such as geometric distortion. We intersect each of these pixel viewing rays with a virtual focal plane (Fig. 6). The intersection points of viewing rays and focal plane form a two-dimensional cloud of measurements, an irregular sampling of the scene (irregular because of parallax and geometric distortion of the channels; Fig. 7). To render an image from this point cloud, we create a regular sampling of the scene by interpolation. For each pixel of the target image, contributions from the nearest measurement points available are added, weighted with the distance from the measurement coordinate to the target pixel (Fig. 8).

Fig. 6

Viewing directions of the pixels of one channel of a multi-aperture system (arrows), intersected with a virtual focal plane (points). The points show how this channel samples object space on a specific focal plane.

Fig. 7

Placing the measurement of multiple channels (four in this case, shown in different colors) on a common focal plane according to the distances of channels and focal plane creates an irregular point cloud.

Fig. 8

To render an image from an irregular point cloud, a regular grid is overlaid on the point cloud. Each of the grid intersections represents a target pixel in the image. Each of the target pixels (black) is calculated by interpolating the nearest measurements from the point cloud (grey).

From the distance $r$, the weight $W_{x,y,i,j}$ of the neighbor $j$, contributing to the target pixel at coordinates $x$ and $y$, is calculated as

The algorithm is presented in full in Ref. 11.

4.2.

Sharpness

Interpolation can be treated as a spatial filter. Calculating the Fourier transform of the filter kernel yields the MTF of the interpolation operation. The interpolation kernel is the continuous version of Eq. (2), the Gaussian

4.3.

Noise

Each target pixel is calculated from the weighted mean of $\nu$ measurements. If a value $V$ is calculated as the weighted sum of measurements $m_{x,y,i,j}$ with equal uncertainties ${\sigma }_{\mathrm{m}}$,

For the following analysis, we first assume uniform density of measurements. In one extreme case, the target pixel is exactly on top of a single source pixel. Choosing a filter width of $w=2$ and setting $r=0$ in Eq. (2), $W_i^{\prime }=1$ (not normalized yet). Four other pixels are at the distance of $r=1$ pixel, yielding $W_i^{\prime }=0.13$. Four further pixels are at a distance of $r=\sqrt{2}=1.4$, yielding $W_i^{\prime }=0.02$. Normalizing yields contributions of $W_i=0.63$, 0.08 and 0.01, respectively. Noise is consequently reduced by a factor of $1/\sqrt{{0.63}^2+4·{0.08}^2+4·{0.01}^2}=1.54$. In the other extreme, the target pixel is exactly between four source pixels, each contributing equally. No other pixels contribute significantly due to their large distance. Noise is decreased by $\sqrt{4}=2$.

In conclusion, noise is decreased by a factor of about 1.54 to 2.

5.1.

Sensitivity

According to theory, the eCLEY should have the same sensitivity as a single-aperture camera with the same aperture F3.7. For verification, we took an image of a uniformly lit target with the image sensor used in the eCLEY, with a single-aperture 16-mm lens (Schneider Cinegon) attached and set to F3.7. The same target was also recorded with an eCLEY. To avoid linearity issues, the exposure time $t_{\exp }$ was adjusted so that both cameras recorded roughly the same mean value (DN) on the target area. The values recorded and the corresponding exposure times $t_{\exp }$ were:

The longer exposure time for the eCLEY suggests a lower sensitivity (by a factor of 0.77). We suspected that this discrepancy is caused by the way the eCLEY objective is attached to the sensor. The clear epoxy filling the gap between objective and substrate has a refractive index close to that of the per-pixel microlenses on the sensor, thereby rendering them ineffective.

We validated our suspicion by attaching a plane glass to one half of the sensor, again filling the air gap with epoxy. We then recorded the same target area with the treated sensor, again imaging the target area with the Cinegon lens set to F3.7. We measured values of 140 on the sensor half without plane glass and 110 on the other half, yielding a relative sensitivity $\gamma =0.71$. This figure also gives us an estimate on the relative area of the photosensor on each pixel. We assume a perfect efficiency of the pixel microlenses and set the fill factor of the sensor pixels to ${\eta }_{\mathrm{pix}}=0.71$.

Sensitivity of the eCLEY consequently has to be adjusted by a factor of 1.40, yielding an adjusted relative sensitivity of about $\gamma =1.1$, higher than the single-aperture lens. The new discrepancy is most likely caused by losses due to internal reflections in the Cinegon lens, which has more air-glass surfaces than the eCLEY objective.

Note that the loss in sensitivity due to the loss of the per-pixel microlenses is not inherent to multi-aperture systems or WLO. The attenuation can be avoided by replacing the bottom substrate with a spacer layer that introduces an air gap between optics module and sensor.

5.2.

Noise

As illustrated in Sec. 4.3, the reconstruction scheme that we use interpolates measurements, which should reduce noise. To verify this claim, we first established the image noise of the sensor used in the eCLEY.

To this end, we recorded 100 images of a scene with a wide dynamic range, using the eCLEY. The recorded images contain microlens images with all values in the dynamic range of the camera, from 0 to 255. As we are interested in temporal noise, we evaluated the temporal behaviour of each pixel. For each of them, the mean and the standard deviation were calculated. Pixels were then distributed into bins of integer values according to their mean. The resulting distribution of standard deviation over image signal is plotted on a log-log scale in Fig. 9.

Fig. 9

Temporal pixel noise (standard deviation) of the sensor in the eCLEY, unprocessed microlens image and reconstructed image (processing filter width $w=2.0$).

We proceeded to process each of the recorded images with our reconstruction algorithm, creating continuous images from the raw images. Filter width $w$ was set to 2.0. These processed frames were characterised pixel by pixel as before, yielding another distribution of standard deviations, this time including reconstruction. This distribution is also plotted in Fig. 9.

Comparing the plots shows that noise is attenuated by a factor of 2.0, being at the top end of our prediction from Sec. 4.3 and validating our model of the reconstruction algorithm.

5.3.

Sharpness

With the results from the previous Secs. 5.1 and 5.2, we have a complete model of the eCLEY transfer function. Figure 10 shows simulations of all components. On top, the diffraction limit for F3.7 is plotted. The optical MTF of the eCLEY central channel is quite close to this limit. It is calculated from a ZEMAX model of the eCLEY objective lens.

Fig. 10

MTF of the eCLEY. From the diffraction limit downwards, one additional component is added to the simulation for each curve. The complete system MTF is plotted at the bottom, with measurements confirming the model at several steps.

Next, the contribution of the sensor is multiplied with the optical MTF. From Sec. 5.1, we assume square photodiodes with a width of $\sqrt{\gamma }·3.2\mu \mathrm{m}=2.7\mu \mathrm{m}$. The crosstalk was modeled as a Gaussian and fitted to results from Refs. 15 and 16.

Finally, the reconstruction step is considered by multiplying the filter kernel $K$ to optical and sensor MTF, with filter width $w=2.0$.

To validate this model, we measured three steps of the image formation process: The optical MTF of a single eCLEY channel, the MTF of a single channel including sensor and the MTF of the complete system, including reconstruction. Each measurement was carried out with the slanted-edge method.¹⁷

The measurements (also plotted in Fig. 10) match the predicted MTFs quite closely, validating our model of the eCLEY. In summary, we have shown that

Note that no calibration was necessary to align the microlens images in the reconstruction step. The distributions of the pixels on the virtual focal plane were taken directly from the ZEMAX model. This fact demonstrates the manufacturing and alignment precision of the microlens array.

Finally, we compared the MTFs of the eCLEY and an OmniVision CameraCube. Figure 11 shows the complete system MTF of both systems. We normalized the frequency axis on the image-space sampling frequency of each camera, which is $1/1.75\mu \mathrm{m}=571\text{cycles}/\mathrm{mm}$ for the CameraCube and $2·1/3.2\mu \mathrm{m}=625\text{cycles}/\mathrm{mm}$ for the eCLEY. The eCLEY exhibits comparable sharpness at reduced total track length.

Fig. 11

MTF of OmniVision CameraCube and eCLEY. System MTFs plotted relative to the sampling frequency of each system.

Figure 12 compares two shots of an USAF test target recorded with the eCLEY and the CameraCube. These photographs also demonstrate similar sharpness for both systems.

Fig. 12

Photographs of a backlit USAF test target with the eCLEY (a) and the OmniVision CameraCube (b). For comparable results, all postprocessing, sharpening and noise reduction was disabled in both cases.

5.4.

Volume

Despite having a larger pixel pitch (3.2 μm instead of 1.75 μm), the eCLEY has a shorter track length than the CameraCube (1.4 mm instead of approximately 2.2 mm). This is the result of $2\times$ supersampling in the eCLEY ($N=2\times 2$, in $x$ and $y$), which cuts total track length in half. Additionally, the eCLEY has only one optical surface per channel instead of the two surfaces of the CameraCube,¹⁸ which also reduces thickness.

Footprint, on the other hand, is larger for the eCLEY, being $6.8\times 5.2\mathrm{mm}$ compared to $3.2\times 2.8\mathrm{mm}$. This $4\times$ increase in footprint is partly due to the larger pixel pitch, partly a result of the segmentation of the FOV ($M=7\times 4$, in $x$ and $y$), as deduced in Sec. 3.4.