2.1

Image normalization and face detection

Image normalization (intensity scaling/stretching) is defined as

where I_N is the normalized image, I₀ is the original image; I_Mm and I_Max are the minimum and maximum pixel value in I₀, while L_Min and L_Max are the desired minimum and maximum pixel value in I_N. For example, we may select L_Min = 0 and ^LMax 255.

Face detection is to localize the face in the image, which is an important step for successive process. Numerous techniques have been developed to detect faces in a single image, e.g., knowledge-based methods, feature invariant facial approaches, template matching methods, appearance-based methods. An excellent survey about face detection can be found in Reference [3]. Many methods use color information, i.e., identify the regions (skin-map) whose color is similar to the color of the skin, which can restrict the searching area significantly. One of the most successful algorithms in visual images is Viola-Jones algorithm⁴ proposed in 2001.

Face detection methods are well developed and quickly mark multiple faces on a picture regardless of their sizes and backgrounds, utilizing spatial changes. The Viola-Jones (VJ) algorithm has become a very common method of object detection, including face detection. Viola and Jones proposed this algorithm as a machine learning approach for object detection with an emphasis on obtaining results rapidly and with high detection rates. The VJ method uses three important aspects. The first is an image representation structure called integral images,⁵’⁶ wherein features are calculated by taking the sum of pixels within multiple rectangular areas. This is of course an extension of a method by Papageorgiou et al.⁷

Viola and Jones admit that the use of rectangles is rather primitive, but they allow for high computational efficiency in calculations.^5,6 This also lends well to the Adaboost algorithm that is used to learn features in the image. This extension of the Adaboost algorithm allows for the system to select a set of features and train the classifiers, a method first discussed by Freund and Schapire.⁸ This learning algorithm allows the system to learn the differences between the integral representations of faces to those of the background.

The last contribution of the VJ method is the use of cascading classifiers. At each stage, the classifier either rejects the instance (represented as a sliding-window from a given test image) based on a given feature value or sends the instance down the tree for more processing. Initially, a large number of negative examples are eliminated. In order to significantly speed the system up, Viola and Jones avoid areas that are highly unlikely to contain the object. The image is tested with the first cascade, and the areas that do not conform to the first cascade are rejected and no longer processed. The areas that may contain the object are further processed until all of the classifiers are tested. The areas in the last classifier are likely to contain the object (face).

2.2

Facial image standardization

We apply general standardization all facial images.

normalized facial image, I_M is the mean image of all faces in the dataset, and is their difference image. I_S is the standardized image; μ and σ denote the mean and standard deviation of the image, respectively.

2.3

Support vector machine model

Support-vector machine (SVM) can efficiently perform a non-linear classification especially with a small dataset.⁹ SVMs are supervised learning models with associated learning algorithms that analyze data used for classification and regression analysis. Given a set of training examples, each marked as belonging to one or the other of two (or multiple) categories, an SVM training algorithm builds a model that assigns new examples to one category or the other, making it a non-probabilistic binary linear.

A SVM constructs a hyperplane or set of hyperplanes in a high- or infinite-dimensional space for classification or regression. Intuitively, a good separation is achieved by the hyperplane that has the largest distance to the nearest training-data point of any class (so-called functional margin), since in general the larger the margin, the lower the generalization error of the classifier.¹⁰ Whereas the original problem defined in a finite-dimensional space may not be linearly separable in that space. For this reason, the original finite-dimensional space can be mapped into a much higherdimensional space, presumably making the separation easier in that space.

Multiclass SVM aims to assign labels to instances by using SVMs, where the labels are drawn from a finite set of several elements. The dominant approach for doing so is to reduce the single multiclass problem into multiple binary classification problems.¹¹

3.1

Convolutional neural network

Convolutional neural networks (CNNs) take a biological inspiration from the visual cortex that has small regions of cells that are sensitive to specific regions of the visual field. CCNs are sort of combination of biology, math and computer science, but these networks have been the most influential innovations in the field of computer vision and artificial intelligence (AI). 2012 was the first year that CNN grew to prominence as Alex Krizhevsky et al¹² used an 8-layer CNN (5 conv., 3 fully-connected) to win that year’s ImageNet competition (referred to as AlexNet thereafter), dropping the classification error record from 25.8% (in 2011) to 16.4% (in 2012), an astounding improvement at the time. Since then many applications have been developed using deep learning.

A common misconception in the deep learning community is that without huge amount of data, it is not possible to create effective deep learning models. While data is a critical part of creating the network, the idea of transfer learning has helped to lessen the data demands. Transfer learning is the process of taking a pretrained model (the weights and parameters of a network that has been trained on a large dataset) and fine-tuning the model with your small dataset. The idea is that the pretrained model will act as a feature extractor. In general, the last layer of the network is replaced with your classification layer (depending on how many classes in your problem). Then the network is trained normally (to get adapted to your dataset).

Let us take a look at the pretrained model with ImageNet, a dataset that contains 14 million images with over 1,000 classes.² The lower layers of the network will detect features like edges and curves. Most likely your network is going to need to detect curves and edges as well. Rather than training the whole network (i.e., all layers) through a random initialization of weights, it is more efficient and effective to use the weights of the pretrained model and focus on the more important layers (higher layers towards classification output) for training. If your dataset is quite different than ImageNet, then you would want to train more of higher layers.

3.2

AlexNet model

AlexNet¹² is a deep CNN for image classification that won the ILSVRC (ImageNet Large-Scale Visual Recognition Challenge)¹³ 2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry. AlexNet has 8 weight layers as shown in Fig. 1. The first 5 are convolutional and the last 3 are fully connected layers. Relu (Rectified Linear Unit) is applied after every convolutional and fully connected layer. Dropout is applied before the first and the second fully connected layer. The input image size should be 227×227×3. The AlexNet has a total of 63M parameters.

Fig. 1

The network architecture of AlexNet¹² (5 conv. + 3 FC layers): conv means convolution, FC means fully connected.

Though there are many network topologies tby six times at thheat have emerged since with lot more layers, AlexNet was the first to make a breakthrough. The AlexNet (1) used Relu instead of Tanh to add non-linearity. It accelerates the speed by six times at the same accuracy; (2) used dropout instead of regularization to deal with overfitting. However the training time is doubled with the dropout rate of 0.5; and (3) overlapped pooling to reduce the size of network. It reduces the top-1 and top-5 error rates by 0.4% and 0.3%, respectively. The AlexNet was trained using batch stochastic gradient descent (SGD), with specific values for momentum and weight decay. The training on two GTX 580 GPUs (for two sub-nets, up and bottom, as shown in Fig. 1) took for about six days. This was the first time a model performed so well on a historically difficult ImageNet dataset, which backed CNNs up with record breaking performance in the competition.

Though there are many network topologies that have emerged since with lot more layers, AlexNet was the first to make a breakthrough. The AlexNet (1) used Relu instead of Tanh to add non-linearity. It accelerates the speed by six times at the same accuacy; (2) used dropout instead of regularization to deal with overfitting. However the training time is doubled with the dropout rate of 0.5; and (3) overlapped pooling to reduce the size of network. It reduces the top-1 and top-5 error rates by 0.4% and 0.3%, respectively. The AlexNet was trained using batch stochastic gradient descent (SGD), with specific values for momentum and weight decay. The training on two GTX 580 GPUs (for two sub-nets, up and bottom, as shown in Fig. 1) took for about six days. This was the first time a model performed so well on a historically difficult ImageNet dataset, which backed CNNs up with record breaking performance in the competition.

3.3

VGG-19 model

Simonyan and Zisserman¹⁴ of the University of Oxford created a 19-layer (16 conv., 3 fully-connected) CNN that strictly used 3×3 filters with stride and pad of 1, along with 2×2 max-pooling layers with stride 2, called VGG-19 model.^14,15 Compared to AlexNet, the VGG-19 (see Fig. 2) is a deeper CNN with more layers. To reduce the number of parameters in such a deep network, it uses small 3×3 filters in all convolutional layers and best utilized with its 7.3% error rate. The VGG-19 model was not the winner of ILSVRC¹³ 2014, however, the VGG Net is one of the most influential papers because it reinforced the notion that CNNs have to have a deep network of layers in order for this hierarchical representation of visual data to work. The motto for CNN architecture design is to keep it deep and keep it simple.

Fig. 2

The network architecture of VGG-19 model¹⁴ (16 conv. + 3 FC layers): conv means convolution, FC means fully connected

The VGG-19 model, a total of 143.7M parameters, was placed 2nd in classification and 1st in localization in ILSVRC 2014. The VGG-19 model is trained on more than a million images (ImageNet) and can classify images into 1000 object categories, for example, keyboard, mouse, pencil, and many animals. As a result, the model has learned rich feature representations for a wide range of images.

3.4

ResNet-50 and Inception-Resnet-v2 models

The ResNet-50 model¹⁶ has 101 layers, which consists of 33 three-layer residual blocks (as shown in Fig. 3a) plus input and output layers. Identity connections learn incremental, or residual, representations, which creates a path for backpropagation. The identity layers gradually transform to complex. Such evolution occurs if the parameters for the f(x) part (shown in Fig. 3a) begin at or near zero. The residual block helps overcome the hard the training problem in DeepNet (> 30 layers) due to vanishing gradients. The ResNet-101 model uses 3×3 filters with stride of 2, and 3×3 max-pooling layers with stride 2. The ResNet-101 was placed the 1st in ImageNet classification in ILSVRC 2015.

Fig. 3.

(a) A three-layer building block for ResNet-50, where the jump (x identity) connection and adder (‘+’) create a path for back-propagation. 256-d means 256 convolutional kernels; (b) Inception Module A in the Inception-v3 model: A large-size kernel is factored into variant small-size convolution kernels.

The Inception-v3^17,18 is a 42-layer CNN model including Inception Module A (see Fig. 3b), B, C. The inception model uses variant kernel size (in v1) to capture the features from variant object size and location, introduces batch (weight) normalization (in v2) and factorizing convolutions (in v3), and uses bottleneck layers (1×1) to avoid a parameter explosion. The Inception-v1 (GoogLeNet) was placed 1st in ImageNet classification in ILSVRC 2014. The Inception-v3 model uses 3 × 3, 5 × 5, 1 × 1 filters with stride 1-2, and 2 × 2 max-pooling layers with stride 2.

The combination of two of the most recent ideas: Residual connections¹⁶ and the latest revised version of the Inception architecture¹⁸. In Ref. [16], it is argued that residual connections are inherently important for training very deep architectures. Since Inception networks tend to be very deep, it is natural to replace the filter concatenation stage of the Inception architecture with residual connections. This would allow Inception to reap the benefits of the residual approach while retaining its computational efficiency. Inception-ResNet-v2 is a hybrid Inception version with residual connections, which leads to dramatically improved recognition performance and training speed in contrast with the Inception architecture.¹⁹

3.5

CNN model summary

Four selected CNN models are summarized in Table 1 and more CNN models are visualized in Fig. 5. All CNN models take color images as inputs. All facial images are scaled to proper size according to each CNN model prior to training and testing. The pretrained CNN models are transferred from the ImageNet.

Fig. 4

Architecture of Inception-ResNet-v2¹⁹: (a) architecture (schema), (b) the stem, (c)-(e) interior inception (35 × 35, 17 × 17 and 8 × 8 grid) modules, (f)-(g) reduction modules.

Fig. 5.

Overview of CNN models: Top-1 accuracy (tested on ImageNet) vs. inference time (a single forward pass). The size of the blobs is proportional to the number of network parameters.

Table 1.

Summary of four CNN models in term of accuracy, layers, parameters, input image, and feature dimension

4.1

Body temperature prediction on three datasets

The performance of body temperature prediction is measured by prediction error and their descriptive statistics shown in Tables 2-4. The descriptive statistics include mean, standard deviation (SD), median, minimum and maximum (range = maximum - minimum).

Table 2.

The prediction error of body temperature (in degrees of Celsius) varying with four CNN-SVM models: Trained on 90% Sony Camera pictures and Tested on 10% Sony camera pictures

The temperature prediction errors on 10% Sony pictures are presented in Table 2. The mean errors approach zero due to training and testing on the same dataset. AlexNet is the best in terms of all descriptive statistics (especially maximum error). Table 3 shows the prediction errors tested with the smartphone pictures. It seems ResNet-50 is the best according to standard deviation and error range. Temperature prediction error increases while tested with video frames, where VGG-19 and ResNet-50 perform very well. It is clear that video-frame images (Fig. 7b) are not as good as digital images (Fig. 6a) in terms of quality (e.g., signal to noise ratio, contrast).

Table 3.

The prediction error of body temperature (in degrees of Celsius) varying with four CNN-SVM models: Trained on 90% Sony Camera pictures and Tested on the Smartphone pictures

Table 4.

The prediction error of body temperature (in degrees of Celsius) varying with four CNN-SVM models: Trained on 90% Sony Camera pictures and Tested on the video frames

Let us walk through one example in Table 3, where testing was conducted on smartphone pictures. Suppose we select ResNet-50-SVM model. The mean predication error is only -0.017 °C with SD = 0.168. With the assumption of normal distribution in all temperature predictions, 95% of prediction errors (2 SD) are within the range: [-0.353, 0.319] °C. This is a reasonable good result considering the size of our datasets.

Overall, AlexNet and ResNet-50 performs very well and reliably, whereas VGG-19 and Inception-ResNet-v2 do not exhibit much advantage.

4.2

Time costs of CNN models

The time costs of CNN feature extraction, SVM model training and testing are given in Table 5. It is clear that AlexNet took the longest time while Inception-ResNet-v2 is the fastest.

Table 5.

Time costs in CNN feature extraction, SVM model training and testing (inference latency) varying with four CNN-SVM models

Facial feature extraction by four CNN models were run on a MSI GT73VR laptop with the following configuration: Intel i7-7820HK CPUs 2.9GHz, 16GB RAM, 1.25TB hard disk, 64-bit Windows 10; NVIDIA GeForce GTX 1070 Graphics Board with 8GB video memory (on board) and 1920 CUDA cores. Four CNN-SVM models were implemented and run in Matlab R2019a (Version 9.6), on a laptop computer, Apple MacBook Pro 2014, with the following configuration: Intel i7-4960HQ CPUs 2.6GHz, 16GB RAM, 1.0TB hard disk, 64-bit Windows 10; NVIDIA GeForce GT 750M Graphics Board with 2GB video memory (on board) and 384 CUDA cores.