Three-dimensional measurement endoscope system with virtual rulers

Hiromasa Nakatani; Keiichi Abe; Atsuo Miyakawa; Susumu Terakawa M.D.

doi:10.1117/1.2800758

1 September 2007 Three-dimensional measurement endoscope system with virtual rulers

Hiromasa Nakatani, Keiichi Abe, Atsuo Miyakawa, Susumu Terakawa M.D.

Author Affiliations +

Journal of Biomedical Optics, Vol. 12, Issue 5, 051803 (September 2007). https://doi.org/10.1117/1.2800758

Abstract

Conventional endoscopic images do not provide quantitative 3-D information. We present an endoscope system that can measure the size and position of an object in real time. Our endoscope contains four laser beam sources and a camera. The procedural steps for 3-D measurements are as follows. First, to obtain the function that maps 2-D coordinates of an image point to its 3-D coordinates in 3-D space, we observe a standard chart with the endoscope lens and determine the correspondence between the image and object height. In addition to the mapping, this function can correct barrel-shaped distortion of endoscopic images. The system detects laser spots on an object surface automatically using a template matching method, and maps the 2-D coordinates of the laser spots to the 3-D coordinates by the triangulation method. Then the system calculates the magnification ratio on the object plane, which is perpendicular to the optical axis and passes the laser spot, so that the system can superimpose a ruler whose scale fits the 3-D coordinates of the object. Thus, physicians can measure the size and position of objects in real time on undistorted images similar to placing rulers on the surface of an organ.

1. Introduction

Endoscopy has been widely used as a minimally invasive diagnostic medical procedure.^{1, 2} Conventional endoscopes with a wide-angle lens can provide physicians with a wide view in the long distance and a magnified view in the short distance. However, since the appearance of an object changes by its distance from the lens, the exact size or location is not obvious only from an endoscopic image. When physicians are required to measure or record the size of the object, they have to thrust a small ruler through a forceps channel in the endoscope and take an internal picture while keeping the ruler within view. This procedure, however, is time consuming even for an experienced endoscopist, and the resulting data may lack precision. Since they can only obtain a rough approximation of the distance, it is difficult for them to judge, for example, whether the size of a tumor is increasing or decreasing. Thus, those methods that can produce an objective 3-D measurement have long been expected.

Recently, novel technologies such as virtual reality and augmented reality are applied to endoscopy and produce 3-D or stereoscopic images using 3-D data from MRI or CT scans.^{3, 4, 5} Regular endoscopes, however, still play a main role in visual inspection and minimally invasive surgery, and 3-D display or measurement that uses only endoscopes is also widely studied. For example, a binocular stereoscopic approach is proposed for increasing the depth and spatial perception of a monocular video endoscope.⁶ 3-D images are also obtained by analyzing the reflection of multicolored light that is projected onto tissue.⁷ There have been proposals to project a laser beam to generate a grid on an object and measure the three dimensions by tracing manually the outline of an object and beam spots,⁸ or to project a striped pattern of light for the 3-D measurements on the stripes.⁹ In this work, we propose an endoscope system that projects multiple laser beams, detects the laser spots automatically, and can display rulers on the endoscopic images, so that physicians can measure the size and position of an object in real time.

2. Methods for Three-Dimensional Measurement

2.1.

System Outline

Our endoscope contains four laser beams, which are directed via optical fibers and set parallel to the optical axis of the camera (Fig. 1 ). The laser beams are projected from the tip of the endoscope onto an object under inspection, and their spots are automatically detected by image processing techniques. Using the 2-D coordinates of those spots, we can calculate the corresponding 3-D coordinate by the triangulation, and can display virtual rulers on the endoscope monitor. At the same time, we can correct barrel distortion associated by the wide angle lens of the endoscope. The procedural steps are as follows.

Fig. 1

Distal end of the laser endoscope.

2.2.

Barrel Distortion Correction

The wide-angle lenses used in regular endoscopes are susceptible to barrel distortions, which can disturb the physicians’ visual perception of shape and size of lesions or suspicious regions.

To remove the distortion from observed images we have to relate the distance of a pixel from the optical axis in an observed image to the corresponding distance in an undistorted image. Once the distortion function is determined, we can use it for all the source images, since the distortion function that gives the correspondences is unique to the particular lens. We observe a test chart shown in Fig. 2a with the endoscope and determine the distortion function by measuring the distances of several points on the distorted image and those on the test chart. Figure 3 shows the resulting distortion function, Fig. 2a shows the source image, and 2b the corrected image.

Fig. 2

Barrel distortion correction (a) distorted and (b) corrected image.

Fig. 3

Distortion function.

2.3.

Laser Spot Detection

To measure the 3-D coordinates of laser spots, the system locates those points in an image using a template matching method.¹⁰ Since the physical configurations of the camera and the laser beams are a priori given, the search areas for the laser spots are limited, and it is not necessary to search the whole image. Since the laser beams of our system are set to be parallel to the optical axis of the camera lens, the search area for the leaser spots are restricted on those lines that are depicted as white lines in Fig. 4 .

Fig. 4

Search area for laser spots.

To determine the location of the laser spots, we compare the density values of each pixel, which are transformed from $8 - bit$ RGB values. To judge the similarity to the laser spot, we measure the density distribution and its deviation in the neighborhood of each pixel. Those pixels that have maximum similarity and larger deviation than a predetermined threshold are identified as the laser spots. The similarity is defined by the correlation between an input image and a Gaussian distribution, since we regard the density profile of the laser spot as Gaussian.

As is shown in our experiments, endoscopic images may have reflection highlights. To prevent the highlights from disturbing the laser spot detection, we regard such pixels that have high values of G or B elements of RGB as highlights and neglect them from the laser spot detection.

2.4.

Calculation of Three-Dimensional Coordinates

Figure 5 illustrates the calculation of the 3-D coordinates of the laser spots. Let the lens optical center $O$ be the origin of the 3-D space, the camera optical axis be the $z$ axis, $L (X_{L}, Y_{L}, Z_{L})$ denote the laser beam source, $P (X_{p}, Y_{p}, Z_{p})$ denote the laser spot on an object, and $p (x_{p}, y_{p}, f)$ denote the laser spot projected on the image plan, where $f$ is the focal length of the lens. Here, for ease of explanation, the image plane is drawn in-between the lens and the object, though it actually is located on the opposite side of the object from the lens.

Fig. 5

Laser spot and ruler.

The 2-D coordinates $(x_{p}, y_{p})$ of the projection on the image plane of the laser spot $(X_{p}, Y_{p}, Z_{p})$ can be calculated after the spot is detected by the method mentioned in Sec. 2.3. Here, the line connecting the origin $O$ and the laser spot $P (X_{p}, Y_{p}, Z_{p})$ is given by

Eq. 1

\frac{x}{x_{p}} = \frac{y}{y_{p}} = \frac{z}{f},

and the line connecting the laser beam source

L (X_{L}, Y_{L}, Z_{L})

and the laser spot

P (X_{p}, Y_{p}, Z_{p})

is given by

Eq. 2

\frac{x - X_{L}}{B_{X}} = \frac{y - Y_{L}}{B_{Y}} = \frac{z - Z_{L}}{B_{Z}},

where

(B_{X}, B_{Y}, B_{Z})

is the direction vector of the laser beam, which is determined when the system is designed. Even if we do not know the value of the direction vector, we can obtain it empirically by measuring the beam spot on a screen that is set at a known distance. In this case, the direction vector can be determined from the position of the spot

(X_{s}, Y_{s}, Z_{s})

:

Eq. 3

(B_{X}, B_{Y}, B_{Z}) = (X_{s} - X_{L}, Y_{s} - Y_{L}, Z_{s} - Z_{L}) .

Then, the laser spot

P (X_{p}, Y_{p}, Z_{p})

is determined as the intersection of Eqs. 1, 2:

4.

X_{p} = \frac{B_{Y} X_{L} - B_{X} Y_{L}}{B_{Y} x_{p} - B_{X} y_{p}} x_{p},

Y_{p} = \frac{B_{Y} X_{L} - B_{X} Y_{L}}{B_{Y} x_{p} - B_{X} y_{p}} y_{p},

Z_{p} = \frac{B_{Y} X_{L} - B_{X} Y_{L}}{B_{Y} x_{p} - B_{X} y_{p}} f .

So far, we have used

p (x_{p}, y_{p}, f)

as the 3-D position of the laser spot on the image plane. It is not easy, however, to measure a point using physical length units on a digital image plane, while it is easy to measure it using pixel units. Even in such a case, we can determine the 3-D coordinates using pixel units instead of physical length units in a way as follows. First, we measure the distance of an arbitrary object and observe it with the endoscope. Suppose the position of the object is

(X_{t}, Y_{t}, Z_{t})

, and the displacement is

t_{t}

pixels in the

x

axis, and

j_{t}

pixels in the

y

axis. Suppose also the physical length of the pixel intervals is

d_{x}

in the

x

axis and

d_{y}

in the

y

axis. Then, the 3-D coordinates of the object can be represented as

(i_{t} d_{x}, j_{t} d_{y}, f)

, and the following equation is obtained:

Eq. 5

\frac{X_{t}}{i_{t} d_{x}} = \frac{Y_{t}}{j_{t} d_{y}} = \frac{Z_{t}}{f} .

The 2-D coordinates of the laser spots

(x_{p}, y_{p})

in Eq. 4 can be represented as

Eq. 6

x_{p} = i_{p} d_{x}, y_{p} = j_{p} d_{y},

where

i_{p}

and

j_{p}

represent the position of the laser beam spots on the image plane when they are measured using pixel units in the

x

and

y

axis, respectively.

Then the 3-D coordinates $(X_{p}, Y_{p}, Z_{p})$ can be determined from Eq. 4 using Eqs. 5, 6:

7.

X_{p} = \frac{B_{Y} X_{L} - B_{X} Y_{L}}{B_{Y} i_{p} j_{t} X_{t} - B_{X} j_{p} i_{t} Y_{t}} i_{p} j_{t} X_{t},

Y_{p} = \frac{B_{Y} X_{L} - B_{X} Y_{L}}{B_{Y} i_{p} j_{t} X_{t} - B_{X} j_{p} i_{t} Y_{t}} j_{p} i_{t} Y_{t},

Z_{p} = \frac{B_{Y} X_{L} - B_{X} Y_{L}}{B_{Y} i_{p} j_{t} X_{t} - B_{X} j_{p} i_{t} Y_{t}} i_{t} j_{t} Z_{t} .

The parameters

x_{p}

,

y_{p}

, and

f

in Eq. 4, which should be measured in physical length units, disappear from Eq. 7, and the 3-D coordinates

(X_{p}, Y_{p}, Z_{p})

can be expressed only by those parameters that can be measured in normal situations.

2.5.

Display of Rulers

Once the 3-D position of the laser spot is measured by the previous method, we can estimate the size of objects near the spot. The system can display rulers virtually on the plane parallel to the image plane, and whose center is at the laser spot (Fig. 5). To depict the divisions of the scale according to its 3-D position, we should calculate the length of the interval of those divisions. Let $u$ be the physical length of the interval, and $u$ is set to $0.5 mm$ in the following examples. Then, we should calculate the length in pixels on the image plane that corresponds to $u$ .

The length of a line segment on the plane $z = Z_{p}$ that passes the laser spot and is parallel to the image plane is reduced by the ratio of $f ∕ Z_{p}$ . The proper interval of the scale divisions is $u f ∕ Z_{p}$ . Even if the focal length $f$ is unknown, the interval of the divisions can be determined from

Eq. 8

\frac{f}{Z_{p}} = \frac{x_{p}}{X_{p}} = \frac{y_{p}}{Y_{p}} = {(\frac{x_{p}^{2} + y_{p}^{2}}{X_{p}^{2} + Y_{p}^{2}})}^{1 ∕ 2},

by the ratio of the distance of the laser spot on the image plane

\sqrt{x_{p}^{2} + y_{p}^{2}}

to the distance of the laser spot from the optical axis

\sqrt{X_{p}^{2} + Y_{p}^{2}}

.

In the same way, if the physical length on the image plane is unknown, the intervals of the divisions can be represented in pixels. In this case, the reduction ratio changes by the direction of the ruler to display. Suppose we should display it in the direction of $θ$ in the real space. The interval of the divisions should be $u f \cos θ ∕ Z_{p} d_{x}$ pixels in the $x$ direction and be $u f \sin θ ∕ Z_{p} d_{y}$ pixels in the $y$ direction. Even if $f ∕ d_{x}$ or $f ∕ d_{y}$ is unknown, we only have to display the divisions by the ratio of the number of pixels to the laser spot on the image to the number of pixels to the spot from the lens optical axis.

In 1Video 1 , four laser beams are projected on the gastric wall of a rat, and the distances to the four spots are measured. Four rulers are also displayed with those divisions whose intervals are calculated according to their distances. The rulers can be displayed in one, two, or four directions, or can be concentric circles.

Video 1

Virtual rulers on laser spots (MPEG, 5.16 MB). 1

10.1117/1.2800758.1

2.6.

Rulers in-between the Laser Beam Spots

When the system places a ruler parallel to the image plane, it marks the divisions repeatedly only with the same interval, since all the divisions of the scale should be equally placed both in the real space and on the image plane. However, a ruler that connects two laser spots is not always parallel to the image plane (Fig. 6 ). Though the divisions are equally placed in the real space, it does not hold for the virtual rulers on the image plane. We have to calculate all the positions of each division.

Fig. 6

Oblique ruler between laser spots.

Fig. 7

Rulers between laser spots.

Suppose that we mark the divisions $M_{k}$ $(k = 0, 1, 2, \dots)$ with the interval of $u$ , starting at a laser spot $P_{s} (X_{s}, Y_{s}, Z_{s})$ and ending at another laser spot $P_{e} (X_{e}, Y_{e}, Z_{e})$ . Then, $(X_{k}, Y_{k}, Z_{k})$ , the 3-D position of the $k$ ’th division $M_{k}$ can be calculated as:

9.

X_{k} = \frac{(∥ P_{e} - P_{s} ∥ - k u) X_{s} + k u X_{e}}{∥ P_{e} - P_{s} ∥},

Y_{k} = \frac{(∥ P_{e} - P_{s} ∥ - k u) Y_{s} + k u Y_{e}}{∥ P_{e} - P_{s} ∥},

Z_{k} = \frac{(∥ P_{e} - P_{s} ∥ - k u) Z_{s} + k u Z_{e}}{∥ P_{e} - P_{s} ∥},

where

∥ P_{e} - P_{s} ∥

represents the distance between two laser spots and can be calculated by the method mentioned in Sec. 2.4. Thus, their corresponding positions on the image plane

m_{k} (x_{k}, y_{k}, z_{k})

can be calculated from

(X_{k}, Y_{k}, Z_{k})

as:

10.

x_{k} = \frac{X_{k}}{Z_{k}} f,

y_{k} = \frac{Y_{k}}{Z_{k}} f,

z_{k} = f .

That means the

k

’th division of the scale should be marked at

Eq. 11

(\frac{X_{k}}{Z_{k}} f, \frac{Y_{k}}{Z_{k}} f),

on the image.

In Eq. 10, the physical length is used in determining the positions of the divisions of the scale. We can also determine those positions if we have to use pixels as the unit. By observing an object with known displacement, as mentioned before, Eq. 5 gives

12.

\frac{f}{d_{x}} = \frac{i_{t} Z_{t}}{X_{t}},

\frac{f}{d_{y}} = \frac{j_{t} Z_{t}}{Y_{t}} .

That means, from the 3-D coordinates

(X_{t}, Y_{t}, Z_{t})

and their projection on the image plane

(i_{t}, j_{t})

in pixels, we can determine the values

f ∕ d_{x}

and

f ∕ d_{y}

. Thus, Eq. 11, i.e., the position of the

k

’th division on the image plane, can be represented in pixels as

(i_{k}, j_{k})

using

d_{x}

and

d_{y}

, the physical length per pixel in the

x

and

y

directions, respectively:

13.

i_{k} = \frac{X_{k} f}{Z_{k} d_{x}} = \frac{X_{k} i_{t} Z_{t}}{Z_{k} X_{t}},

j_{k} = \frac{Y_{k} f}{Z_{k} d_{y}} = \frac{Y_{k} j_{t} Z_{t}}{Z_{k} Y_{t}} .

Therefore, we only have to mark the

k

’th division at

(\frac{X_{k} i_{t} Z_{t}}{Z_{k} X_{t}}, \frac{Y_{k} j_{t} Z_{t}}{Z_{k} Y_{t}}),

in pixels.

Thus we can depict oblique rulers between the spots. Figure 7 shows an example of displaying the rulers in-between the four laser spots.

3. Experiments

We have modified a commercially available endoscope Fujinon EC-450WM5, which is used for the lower gastro-intestinal (GI) tract, with a $140 - \deg$ view field and $12.8 - mm$ distal end diameter. The diameter of the optical fibers, via which the four laser beams are directed, is $120 μ m$ . The maximum power of the laser beam is $6 mW$ , and the wavelength is set to $635 nm$ to lessen the beam absorption into tissues.

To evaluate the practicability, we used the proposed system to observe the gastric wall of a rat. The figures shown in Sec. 2 were taken during one of those experiments. To estimate the detection rate of the laser spot, we checked 1000 consecutive frames and found that the rate of correct detection was 85%. Most failures occurred when the system mistook reflection highlights for laser spots. It does not matter, however, when it comes to clinical applications, because physicians do not take the trouble to see highlighted areas to observe all the spot locations in each and every frame, but they only need their locations of interest. In addition, our system is made interactive, so that physicians press the foot switch only when they need 3-D measurements while seeing the motion of the laser spots. The detection rate increases to 97% for those areas without highlights.

The processing time for each frame with an Intel Xeon 5160, $3.00 GHz \times 2$ is as follows: $9.6 \times 10^{- 3} \sec$ for the distortion correction, $6.3 \times 10^{- 3} \sec$ for the laser spot detection, and $1.0 \times 10^{- 6} \sec$ for the 3-D coordinate calculation and ruler display. Thus, the total processing time for a frame is $1.6 \times 10^{- 2} \sec$ , so that all the procedures can be performed in real time.

To estimate the accuracy of the distance measurements, we observed a screen at a distance varying between 5 and $40 mm$ . We used the parameters obtained at $20 mm$ to measure the other distances in such a way as mentioned in Sec. 2.4, and found the differences between the distances that the system calculated and the actual ones are within the range of $0.2 mm$ . We also estimated the accuracy of the virtual rulers by observing graph paper at a distance varying between 5 and $40 mm$ . We compared the actual lengths on the graph paper and the lengths measured with the virtual ruler, and found that the average error in the measurement is 5.1%.

4. Conclusions

We present a laser beam endoscope system for 3-D measurements in real time. The system detects the laser spots automatically and calculates their 3-D coordinates by the triangulation method. Physicians can see the rulers that fit the 3-D position, and can measure the size and position of objects on undistorted images, similar to placing rulers on the surface of an organ.

Though we cannot use the proposed system for bedside diagnosis because it needs official medical device evaluation, the experiments using rats show the effectiveness of the system for clinical use.

References

1.

R. B. Northrop, Noninvasive Instrumentation and Measurement in Medical Diagnosis, (2002) Google Scholar

2.

A. Katzir, Lasers and Optical Fibers in Medicine, (1993) Google Scholar

3.

B. J. Wood and P. Razavi, “Virtual endoscopy: a promising new technology,” Am. Fam. Physician, 66 (1), 107 –112 (2002). 0002-838X Google Scholar

4.

A. del Rio, D. Bartz, R. Jäger, Ö. Gürvit, and D. Freudenstein, “Efficient stereoscopic rendering in virtual endoscopy applications,” 95 –101 (2003). Google Scholar

5.

F. Devernay, F. Mourgues, and È. Coste-Manière, “Towards endoscopic augmented reality for robotically assisted minimally invasive cardiac surgery,” 16 –20 (2001). Google Scholar

6.

T. Thormählen, H. Broszio, and P. N. Meier, “Three-dimensional endoscopy,” Medical Imaging in Gastroenterology and Hepatology, Kluwer Academic Publishers, New York (2002). Google Scholar

7.

D. Yelin, I. Rizvi, W. M. White, J. T. Motz, T. Hasan, B. E. Bouma, and G. J. Tearney, “Three-dimensional miniature endoscopy,” Nature (London), 443 (19), 765 (2006). https://doi.org/10.1038/443765a 0028-0836 Google Scholar

8.

M. Yamaguchi, Y. Okazaki, H. Yanai, and T. Takemoto, “Three-dimensional determination of gastric ulcer size with laser endoscopy,” Endoscopy, 20 (5), 263 –266 (1988). 0013-726X Google Scholar

9.

K. Hasegawa and Y. Sato, “Endoscope system for high-speed 3D measurement,” Syst. Comput. Japan, 32 (8), 30 –39 (2001). 0882-1662 Google Scholar

10.

R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification, 2nd ed.John Wiley and Sons, New York (2001). Google Scholar

Citation Download Citation

Hiromasa Nakatani, Keiichi Abe, Atsuo Miyakawa, and Susumu Terakawa M.D. "Three-dimensional measurement endoscope system with virtual rulers," Journal of Biomedical Optics 12(5), 051803 (1 September 2007). https://doi.org/10.1117/1.2800758

Published: 1 September 2007

Access the abstract

JOURNAL ARTICLE
6 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

CITATIONS

Cited by 15 scholarly publications and 1 patent.

Explore citations on Lens.org

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Endoscopes

3D image processing

Distortion

3D displays

3D metrology

Endoscopy

Cameras

1.

Introduction

2.

Methods for Three-Dimensional Measurement

2.1.

System Outline

Fig. 1

2.2.

Barrel Distortion Correction

Fig. 2

Fig. 3

2.3.

Laser Spot Detection

Fig. 4

2.4.

Calculation of Three-Dimensional Coordinates

Fig. 5

Eq. 1

Eq. 2

Eq. 3

4.

Eq. 5

Eq. 6

7.

2.5.

Display of Rulers

Eq. 8

Video 1

2.6.

Rulers in-between the Laser Beam Spots

Fig. 6

Fig. 7

9.

10.

Eq. 11

12.

13.

3.

Experiments

4.

Conclusions

References

Show All Keywords

Keywords/Phrases

Search In:

Publication Years