1.

Introduction

The grayscale frames interpolation problem is the process of creating intermediary grayscale frames between two given frames.¹ The interpolated grayscale frame $\left(X_i\right)$ , $0⩽i⩽M$ , $M⩾2$ depends on the content of an input grayscale frame and an output grayscale frame. For $i=0$ , the interpolated grayscale frame is equal to the input grayscale frame $X_0$ ; for $i=M$ , the interpolated grayscale frame is equal to the output grayscale frame $X_M$ .²

The simplest method involves linear interpolation.^{3, 4} In Ref. 5, a dynamically elastic surface interpolation scheme was proposed. A hybrid approach that combined elastic interpolation, spline theory, and the surface consistency theorem was proposed to produce further improvement.^{6, 7} Grevera and Udupa⁸ and Herman ⁹ interpolated the distance and proposed shape-based methods by encoding the segmented image with distance codes. Guo developed a morphology-based interpolation method and successfully resolved the problems in objects with holes and large offsets.¹⁰ Lee proposed another morphology-based scheme that was simpler in computational complexity.¹¹ Goshtasby selected feature points from successive frames to control the gray-level interpolation.¹² In Ref. 13, feature points were used that were based on a fuzzy measure of the boundaries and medial axis transforms. In Ref. 14, a morphological skeleton interpolation was proposed in which interpolated sets were generated from a succession of skeletons derived from the matching of two neighboring set skeletons. Berier and Neely computed the interpolation with feature line-segment control.¹⁵ An example of object-based methods that used features is shape-based interpolation,¹⁶ which was extended by allowing registration between frames¹⁷ and using feature guidance.¹⁸ For grayscale images, mathematical morphology provides a well-founded theory.¹⁹

The new interpolation grayscale frame method presented in this paper is based on the morphological $S$ transform (MST) and consists of partially changing, step by step using growing structuring elements (SEs) the morphological $S$ transforms of the foreground content of an input grayscale frame with the MSTs of the foreground content of an output grayscale frame. Better performance comes at the expense of not very great computational complexity.

This paper is organized as follows. Section 2 defines the grayscale MST. Section 3 presents the grayscale frames interpolation algorithm. Section 4 provides experimental results. The conclusions of this study are drawn in Sec. 5.

2.

Morphological $S$ Transform (MST)

The MST of a grayscale image $X$ can be calculated by means of morphological operations.¹⁹ For any SE $B$ , we have

The morphological $S(X,nB)$ transform of a grayscale image $X$ by SE $nB$ is given by

A grayscale image can be represented using the morphological formula

3.

Interpolation Method

The new and improved grayscale frame interpolation method presented in this paper consists of partially changing, step by step using growing SEs $nB$ for $n=N,\dots ,1,0$ with $0B=(0,0)$ , the MST of the foreground content of a grayscale input frame with the MST of the foreground content of an output grayscale frame.

Based on the grayscale input frame $X_0$ and grayscale output frame $X_M$ , the foreground contents are defined as

The interpolated foreground content $X_i^f$ of grayscale interpolated frame $X_i$ , for $1⩽i⩽N$ and $N⩾2$ for $N=\max (N_0,N_M)$ , obtained by using the morphological formula

Using morphological formula 7, we can build a large number, $\min (N_0,N_M)$ , of not necessarily different interpolated grayscale frames.

4.

Experimental Results

We used the proposed algorithm for grayscale frame interpolation by constructing a frame using growing SEs and different $i$ values. We performed the experiments using grayscale images.

An example is presented in Fig. 1 . Test input and output grayscale frames are presented in Figs. 1 and 1, and the “missing” frame $X_{\mathrm{ref}}$ is in Fig. 1. This frame will be used as reference to compare our results. Our frames have dimensions [256, 256].

Fig. 1

Input, output, and reference grayscale frames for the interpolation example: (a) grayscale input frame $X_i$ , (b) grayscale output frame $X_M$ , and (c) grayscale reference frame $X_{\mathrm{ref}}$ .

Using formula 5, we obtained the foreground contents for input and output frame $X_0^f$ and $X_M^f$ for input and output frame $X_0$ and $X_M$ [Figs. 2 and 2 ]. The background content for both the input and output frames (content the frames have in common), $X^b$ [Fig. 2], is given by

Fig. 2

Foreground and background contents for the input frame and output frames: (a) foreground contents for input frame $X_0^f$ , (b) foreground contents for output frame $X_M^f$ , and (c) background content of both frames (content the frames have in common) $X^b$ .

Using the morphological $S$ transform and crosses or squares as SE $B$ , we obtained $N_0^c=38$ , $N_M^c=38$ , and $N^c=38$ for the crosses and $N_0^s=30$ , $N_M^s=30$ , and $N^s=30$ for the squares. Interpolated foreground content $X_i^f$ , with $1⩽i⩽38$ (for the crosses) and $1⩽i⩽30$ (for the squares), was obtained using morphological formula 7. Finally, grayscale interpolated frame $X_i$ was given by formula 8. For the crosses, we built 34 interpolated grayscale frames, but only 21 were different. For the squares, we built 30 interpolated grayscale frames, but only 17 were different. Using a standard computer and an optimized C routine implementation, we obtained a grayscale interpolated frame at $40\mathrm{ms}$ . For $\text{size}\left(X_{\mathrm{ref}}\right)=\left[AB\right]$ , we can define error function $r_i$ by

Some error function results for the crosses $\left(r_i^c\right)$ and squares $\left(r_i^s\right)$ SEs are presented in Tables 1, 2 .

Table 1

Error function ric .

Table 2

Error function ris .

Interpolated frames using different SEs are presented in Figs. 3 and 4 .

Fig. 3

Interpolated frame ( $X_i$ , $i=12$ ) using crosses as the SE.

Fig. 4

Interpolated frame ( $X_i$ , $i=10$ ) using squares as the SE.

After simulations using different SEs and different $i$ values, we observed that one of the interpolated frames offered by our method was similar to the original reference frame.

5.

Conclusions

This paper has addressed the grayscale frames generalized interpolation by means of mathematical morphology.

After describing the algorithm, we provided experimental results that were very encouraging. We applied the proposed scheme to real-world data. For many situations, the proposed scheme was experimentally shown to successfully resolve complex interpolation problems. Some preprocessing could improve the results. The algorithm is fully morphological and can be applied quickly. This entire process is efficient without significant computational complexity. However, it is difficult to state rigorously that our method produces theoretically correct interpolated results for any frames.