Efficient content-based image retrieval (CBIR) of biomedical images is a challenging problem. Feature representation algorithms used in indexing medical images on the pathology of interest have to address conflicting goals of reducing feature dimensionality while retaining important and often subtle biomedical features. In case of the vertebra, its shape effectively describes various pathologies identified by medical experts as being consistently and reliably found in x-rays in the image collection. A suitable shape method must enable retrieval relevant to the pathology in question. An approach to enabling pathology based retrieval is to use partial shape matching techniques. This paper describes our research in the development of such methods and initial retrieval results and related issues. The research is a part of our ongoing effort in developing CBIR for digitized images of a collection of 17,000 cervical and lumbar spine x-rays taken as a part of the second National Health and Nutrition Examination Survey (NHANES II) at the Lister Hill National Center for Biomedical Communications, an intramural R&D division of the U.S. National Library of Medicine.

To allow efficient browsing of large image collections, we have to provide a summary of its visual content. We present in this paper a robust approach to organize image databases: the Adaptive Robust Competition (ARC). This algorithm relies on a non-supervised database categorization, coupled with a selection of prototypes in each resulting category. This categorization is performed using image descriptors, which describe the visual appearance of the images. A principal component analysis is performed for every feature to reduce dimensionality. Then, clustering is performed in challenging conditions by minimizing a Competitive Agglomeration objective function with an extra noise cluster to collect outliers. The competition is improved to be adaptive to clusters of various densities. In a second step, we provide the user with tools to correct possible misclassifications and personalize the image categories. The constraints to deal with for such a system are the simplicity of the user feedback and the rapidity to propose a new category based on the user's criteria.

The paper proposes an adaptive retrieval approach based on the concept of relevance-feedback. The proposed approach establishes a link between high-level concepts and low-level features, using the user's feedback not only to assign proper weights to the features, but also to dynamically select them within a large collection of parameters. The target is to identify the set of relevant features according to a user query, maintaining at the same time a small sized feature vector to attain better matching and lower complexity. To this end, after each feedback the image description is modified by removing less significant features and better specifying most significant ones. The weights are then adjusted based on relevant and irrelevant output images without further requiring the user intervention. Results achieved on a large set of synthetic and natural data show that the proposed algorithm outperforms previously proposed methods.

In this paper a novel "Smart Media" concept for semantic-based multimedia security and management is proposed. This concept is based on interactive object segmentation (considered as side information in visual human-computer interface) with hidden object-based annotations. Information-theoretic formalism is introduced that considers the human-computer interface as a multiple access channel. We do not consider an image as a set of pixels but rather as a set of annotated regions that correspond to objects or their parts, where these objects are associated with some hidden descriptive text about their features. The presented approach for "semantic" segmentation is addressed by means of the human-computer interface that allows a user to easily incorporate information related to image objects and to store them in a secure way. Since each selected image object carries its own embedded description, this makes it self-containing and formally independent from the particular image format used for storage in image databases. The proposed object-based hidden descriptors are invariant to changes of image filename or/and image headers, and are resistant to object cropping/insertion operations, which are usual in multimedia processing and management. This is well harmonized with the "Smart Media" concept where the image contains additional information about itself, and where this information is securely integrated inside the image while remaining perceptually invisible.

Accurate image segmentation is important for many image, video and computer vision applications. Over the last few decades, many image segmentation methods have been proposed. However, the results of these segmentation methods are usually evaluated only visually, qualitatively, or indirectly by the effectiveness of the segmentation on the subsequent processing steps. Such methods are either subjective or tied to particular applications. They do not judge the performance of a segmentation method objectively, and cannot be used as a means to compare the performance of different segmentation techniques. A few quantitative evaluation methods have been proposed, but these early methods have been based entirely on empirical analysis and have no theoretical grounding. In this paper, we propose a novel objective segmentation evaluation method based on information theory. The new method uses entropy as the basis for measuring the uniformity of pixel characteristics (luminance is used in this paper) within a segmentation region. The evaluation method provides a relative quality score that can be used to compare different segmentations of the same image. This method can be used to compare both various parameterizations of one particular segmentation method as well as fundamentally different segmentation techniques. The results from this preliminary study indicate that the proposed evaluation method is superior to the prior quantitative segmentation evaluation techniques, and identify areas for future research in objective segmentation evaluation.

In the domain of video indexing, one of the research topics is the automatic extraction of information to reach the objective of automatically describing and organizing the content. Thinking of a video stream, different kinds of information can be taken into account, but we can suppose that most of the information is contained in the foreground objects so that number of objects, their shape, their contours and so on, can constitute a good guess for the content description. This paper describes a new approach to extract foreground objects in MPEG2 video stream, in the framework of "rough indexing paradigm" we define. This paradigm leads us to reach the purpose in near real time, nevertheless maintaining a good level of details.

Video segmentation, storage and content-based retrieval become essential with the development of government affairs through network and web search. Segmentation and clustering of the video sequences are the first step should be taken to solve the problem. The conventional segmentation methods are based on histogram difference and pixel difference between the successive frame pairs, which usually can not show satisfying results. In this paper we make use of the mutual complementary features between the pixel gray distribution based method and the object outline based method throughout the frames to segment the video sequence hierarchically.

We propose an automatic camera calibration algorithm for court sports. The obtained camera calibration parameters are required for applications that need to convert positions in the video frame to real-world coordinates or vice versa. Our algorithm uses a model of the arrangement of court lines for calibration. Since the court model can be specified by the user, the algorithm can be applied to a variety of different sports. The algorithm starts with a model initialization step which locates the court in the image without any user assistance or a-priori knowledge about the most probable position. Image pixels are classified as court line pixels if they pass several tests including color and local texture constraints. A Hough transform is applied to extract line elements, forming a set of court line candidates. The subsequent combinatorial search establishes correspondences between lines in the input image and lines from the court model. For the succeeding input frames, an abbreviated calibration algorithm is used, which predicts the camera parameters for the new image and optimizes the parameters using a gradient-descent algorithm. We have conducted experiments on a variety of sport videos (tennis, volleyball, and goal area sequences of soccer games). Video scenes with considerable difficulties were selected to test the robustness of the algorithm. Results show that the algorithm is very robust to occlusions, partial court views, bad lighting conditions, or shadows.

This paper proposes a novel sequence matching technique to detect copies of a video clip. This copy detection scheme can be used as either an alternative approach or a complementary approach to watermarking for copyright protection. It can also be used for media tracking. A challenge is that the different digitizing and encoding processes give rise to several distortions, such as changes in brightness, changes in color, different blocky artifacts, changes in frame format, and so on. While there exist several algorithms proposed, most of them focus on coping with signal distortions introduced by different encoding parameters. However, we note techniques to deal with aspect ratio conversions must be evenly taken into account because those conversions are frequent in practice. To this end, each image frame is partitioned into 2x2 by intensity averaging, and the partitioned values are stored for indexing and matching. Our spatio-temporal approach combines spatial matching of ordinal signatures obtained from the partitions of each frame and temporal matching of temporal signatures from the temporal trails of the partitions. The efficacy of the proposed method has been extensively tested and the results show the proposed scheme is efficient to deal with various modifications.

In this paper, we propose a home video indexing using an audio information to detect an event both a rules-based method and a GMM-based method. Although exclusive audio segmentation and classification was usually used, various sounds overlap in practice, in which case an audio in which various sound overlapped is expressed by a labeling layered index. With the rules-based method, low-level audio features are used to determine indexes, which are classied such as speech, silence, music, and EVN(Environment Noise). The GMM-based method which uses the same features as the rule based method also classifies an audio into the four classes. Smoothing is applied in order to determine the index. We show experiments in a few home video data.

Segmenting audio data into the smallest musical components is the basis for many further meta data extraction algorithms. For example, an automatic music transcription system needs to know where the exact boundaries of each tone are. In this paper a note accurate audio segmentation algorithm based on MPEG-7 low level descriptors is introduced. For a reliable detection of different notes, both features in the time and the frequency domain are used. Because of this, polyphonic instrument mixes and even melodies characterized by human voices can be examined with this alogrithm. For testing and verification of the note accurate segmentation, a simple music transcription system was implemented. The dominant frequency within each segment is used to build a MIDI file representing the processed audio data.

The production of closed captions is an important but expensive process in video broadcasting. We propose a method to generate highly accurate off-line captions efficiently. Our system uses text alignment to synchronize program transcripts obtained for a video program with text produced by an automatic speech recognition (ASR) system. We will also describe the accuracy in both closed-caption text and the ASR output for a number of news programs and provide a detailed analysis of the errors that occur.

Realtime music identification became more and more interesting within the past few years. Possible fields are for example monitoring a radio station in order to create a playlist or scanning network traffic in search of copyright protected material. This paper presents a client-server application to identify an unknown segment of music. The extraction and exchange of descriptive data is done with MPEG-7 only. This paper also explains how to define the similarity between two segments of music and determents its robustness towards perceptional audio coding and filtering. It also introduces an indexing system to reduce the number of segments which have to be compared to the query.

The term "color temperature" usually represents the color of light source or the white point of image displaying devices. The color temperature can be an effective bridge between images' characteristic and human's perceptual temperature feeling against the images. It can capture human's high-level perception to improve image browsing. In this paper, our goal is to demonstrate how well the color temperature is connected to such human's perception. We demonstrate the method of subjective experiment, a color temperature mapping range for each perceptual category, and browsing accuracy of the color temperature ranges obtained from the experiment. The compact representation for the color temperature and some of usage scenarios are explained as presented in the amendment of MPEG-7 standard.

Visual information retrieval (VIR) is a research area with more than 300 scientific publications every year. Technological progress lets surveys become out of date within a short duration. This paper intends to shortly describe selected important advances in VIR in recent years and point out promising directions for future research. A software architecture for visual media handling is proposed that allows handling image and video content equally. This allows to integrate both types of media in a singe system. The major advances in feature design are sketched and new methods for semantic enrichment are proposed. Guidelines are formulated for further development of feature extraction methods. The most relevant retrieval processes are described and an interactive method for visual mining is suggested that really puts "the human in the loop". For evaluation, the classic recall- and precision-based approach is discussed as well as a new procedure based on MPEG-7 and statistical data analysis. Finally, an "ideal" architecture for VIR systems is outlined. The selection of VIR topics is subjective and represents the author's point of view. The intention is to provide a short but substantial introduction to the field of VIR.

Evaluation in visual information retrieval is usually performed by executing test queries and calculating recall and precision based on predefined media collections and ground truth information. This process is complex and time consuming. For the evaluation of feature transformations (transformation of visual media objects to feature vectors) it would be desirable to have simpler methods available. In this paper we introduce an evaluation procedure for features that is based on statistical data analysis. The new idea is that we make use of the existing visual MPEG-7 descriptors to judge the characteristics of novel feature transformations. The proposed procedure is divided into four steps: (1) feature extraction, (2) merging with MPEG-7 data and normalisation, (3) statistical data analysis and (4) visualisation and interpretation. Three types of statistical methods are used for evaluation: (1) description (moments, etc.), (2) identification of similarities (e.g. cluster analysis) and (3) identification of dependencies (e.g. factor analysis). From statistical analysis several benefits can be drawn for feature redesign. Application of the evaluation procedure suggested and advantages of the approach are shown in several examples.

Moving objects are complicated to manage because they involve temporal attributes as well as spatial attributes. There are two methods to represent the motion of moving objects, function method and sampling method. Motion state modeling, based on sampling method, can give object's position, orientation and their changes at a specific epoch, and encapsulates all the calculation by object orientation method. A big job is to search the motion state vectors efficiently, which can be performed with the help of 2ⁿ index trees. 2ⁿ index tree is an efficient index method to multi-dimensional data. Different kinds of motion data retrieval can be transformed to basic searching in 2ⁿ index trees. With proper operation algorithm, 2ⁿ index trees work well with the indexing and retrieval of moving objects.

Video clip retrieval is a significant research topic of content-base multimedia retrieval. Generally, video clip retrieval process is carried out as following: (1) segment a video clip into shots; (2) extract a key frame from each shot as its representative; (3) denote every key frame as a feature vector, and thus a video clip can be denoted as a sequence of feature vectors; (4) retrieve match clip by computing the similarity between the feature vector sequence of a query clip and the feature vector sequence of any clip in database. To carry out fast video clip retrieval the index structure is indispensable. According to our literature survey, S²-tree [17] is the one and only index structure having been applied to support video clip retrieval, which combines the characteristics of both X-tree and Suffix-tree and converts the series vectors retrieval to string matching. But S²-tree structure will not be applicable if the feature vector's dimension is beyond 20, because the X-tree itself cannot be used to sustain similarity query effectively when dimensions of vectors are beyond 20. Furthermore, it cannot support flexible similarity definitions between two vector sequences. VA-file represents the vector approximately by compressing the original data and it maintains the original order when representing vectors in a sequence, which is a very valuable merit for vector sequences matching. In this paper, a new video clip similarity model as well as video clip retrieval algorithm based on VA-File are proposed. The experiments show that our algorithm incredibly shortened the retrieval time compared to sequential scanning without index structure.

In this paper, we present a classification and retrieval technique targeted for retrieval of home video abstract using dimension-reduced, decorrelated spectral features of audio content. The feature extraction based on MPEG-7 descriptors consists of three main stages: Normalized Audio Spectrum Envelope (NASE), basis decomposition algorithm and basis projection, obtained by multiplying the NASE with a set of extracted basis functions. A classifier based on continuous hidden Markov models is applied. For retrieval with accurate performance the system consists of a two-level hierarchy method using speech recognition and sound classification. For the measure of the performance we compare the classification results of MPEG-7 standardized features vs. Mel-scale Frequency Cepstrum Coefficients (MFCC). Results show that the MFCC features yield better performance compared to MPEG-7 features.

In classic pattern recognition problems, classes are mutually exclusive by definition. Classification errors occur when the classes overlap in the feature space. We examine a different situation, occurring when the classes are, by definition, not mutually exclusive. Such problems arise in scene and document classification and in medical diagnosis. We present a framework to handle such problems and apply it to the problem of semantic scene classification, where a natural scene may contain multiple objects such that the scene can be described by multiple class labels (e.g., a field scene with a mountain in the background). Such a problem poses challenges to the classic pattern recognition paradigm and demands a different treatment. We discuss approaches for training and testing in this scenario and introduce new metrics for evaluating individual examples, class recall and precision, and overall accuracy. Experiments show that our methods are suitable for scene classification; furthermore, our work appears to generalize to other classification problems of the same nature.

The performance of an exemplar-based scene classification system depends largely on the size and quality of its set of training exemplars, which can be limited in practice. In addition, in non-trivial data sets, variations in scene content as well as distracting regions may exist in many testing images to prohibit good matches with the exemplars. We introduce the concept of image-transform bootstrapping using image transforms to address such issues. In particular, three major schemes are described for exploiting this concept to augment training, testing, and both. We have successfully applied it to three applications of increasing difficulty: sunset detection, outdoor scene classification, and automatic image orientation detection. It is shown that appropriate transforms and meta-classification methods can be selected to boost performance according to the domain of the problem and the features/classifier used.

Image classification can facilitate semantic retrieval and browsing of large-scale image databases. Existing approaches are usually based on extracting local or global low-level features such as color, edge, and texture from images. In this paper, we propose an image categorization method that characterizes the respective scene structures in images. 2D Spatial Frequency Map of an image, as well as the respective projection vector representations and principal component representations, are used to characterize the spatial structure of the image. Based on multiple similarity scores, we use a spectral clustering method and a maximal-spanning-tree-spectral-clustering method to generate image categories.

In this paper, a music segmentation framework is proposed to segment music streams based on human perception. In the proposed framework, three perceptual features corresponding to four perceptual properties are extracted. By analyzing the trajectory of feature values, the cutting points of a music stream can be identified. According to the complementary characteristics of the three features, a ranking algorithm is designed to achieve a better accuracy. We perform a series of experiments to evaluate the Complementary Characteristics and the effectiveness of the proposed framework.

We discuss the meaning and significance of the video mining problem, and present our work on some aspects of video mining. A simple definition of video mining is unsupervised discovery of patterns in audio-visual content. Such purely unsupervised discovery is readily applicable to video surveillance as well as to consumer video browsing applications. We interpret video mining as content-adaptive or "blind" content processing, in which the first stage is content characterization and the second stage is event discovery based on the characterization obtained in stage 1. We discuss the target applications and find that using a purely unsupervised approach are too computationally complex to be implemented on our product platform. We then describe various combinations of unsupervised and supervised learning techniques that help discover patterns that are useful to the end-user of the application. We target consumer video browsing applications such as commercial message detection, sports highlights extraction etc. We employ both audio and video features. We find that supervised audio classification combined with unsupervised unusual event discovery enables accurate supervised detection of desired events. Our techniques are computationally simple and robust to common variations in production styles etc.

In this paper, we present our new results in news video story segmentation and classification in the context of TRECVID video retrieval benchmarking event 2003. We applied and extended the Maximum Entropy statistical model to effectively fuse diverse features from multiple levels and modalities, including visual, audio, and text. We have included various features such as motion, face, music/speech types, prosody, and high-level text segmentation information. The statistical fusion model is used to automatically discover relevant features contributing to the detection of story boundaries. One novel aspect of our method is the use of a feature wrapper to address different types of features -- asynchronous, discrete, continuous and delta ones. We also developed several novel features related to prosody. Using the large news video set from the TRECVID 2003 benchmark, we demonstrate satisfactory performance (F1 measures up to 0.76 in ABC news and 0.73 in CNN news), present how these multi-level multi-modal features construct the probabilistic framework, and more importantly observe an interesting opportunity for further improvement.

This paper describes a system for the mining of surveillance video. Our main contributions are: providing a high level query language for submitting queries about spatial and temporal relations of background regions and moving entities, and about human activities; providing a compiler to map high level queries into a set of novel Petri net filters that utilize computer vision algorithms to answer components of the queries; and providing a powerful graphical interface where users have the ability to formulate the query visually.

Model vector-based retrieval is a novel approach for video indexing that uses a semantic model vector signature that describes the detection of a fixed set of concepts across a lexicon. The model vector basis is created using a set of independent binary classifiers that correspond to the semantic concepts. The model vectors are created by applying the binary detectors to video content and measuring the confidence of detection. Once the model vectors are extracted, simple techniques can be used for searching to find similar matches in a video database. However, since confidence scores alone do not capture information about the reliability of the underlying detectors, techniques are needed to ensure good performance in the presence of varying qualities of detectors. In this paper, we examine the model vector-based retrieval framework for video and propose methods using detector validity to improve matching performance. In particular, we develop a model vector distance metric that weighs the dimensions using detector validity scores. In this paper, we explore the new model vector-based retrieval method for video indexing and empirically evaluate the retrieval effectiveness on a large video test collection using different methods of measuring and incorporating detector validity indicators.

This paper presents an overview of our recent work on managing image and video data. The first half of the paper describes a representation for the semantic spatial layout of video frames. In particular, Markov random fields are used to characterize the spatial arrangement of frame tiles that are labeled using support vector machine classifiers. The representation is shown to support similarity retrieval at the semantic level as demonstrated in a prototype video management system. The second half of the paper describes a method for efficiently computing nearest neighbor queries in high-dimensional feature spaces in a relevance feedback framework.

Removing commercials from television programs is a much sought-after feature for a personal video recorder. In this paper, we employ an unsupervised clustering scheme (CM_Detect) to detect commercials in television programs. Each program is first divided into W8-minute chunks, and we extract audio and visual features from each of these chunks. Next, we apply k-means clustering to assign each chunk with a commercial/program label. In contrast to other methods, we do not make any assumptions regarding the program content. Thus, our method is highly content-adaptive and computationally inexpensive. Through empirical studies on various content, including American news, Japanese news, and sports programs, we demonstrate that our method is able to filter out most of the commercials without falsely removing the regular program.

We present an extension to earlier work on recognizing logos in video stills. The logo instances considered here are rigid planar objects observed at a distance in the scene, so the possible perspective transformation can be approximated by an affine transformation. For this reason we can classify the logos by matching (invariant) line profiles. We enhance our previous method by considering multiple line profiles instead of a single profile of the logo. The positions of the lines are based on maxima in the Hough transform space of the segmented logo foreground image. Experiments are performed on MPEG1 sport video sequences to show the performance of the proposed method.

We propose a video genre classification method using multimodal features. The proposed method is applied for the preprocessing of automatic video summarization or the retrieval and classification of broadcasting video contents. Through a statistical analysis of low-level and middle-level audio-visual features in video, the proposed method can achieve good performance in classifying several broadcasting genres such as cartoon, drama, music video, news, and sports. In this paper, we adopt MPEG-7 audio-visual descriptors as multimodal features of video contents and evaluate the performance of the classification by feeding the features into a decision tree-based classifier which is trained by CART. The experimental results show that the proposed method can recognize several broadcasting video genres with a high accuracy and the classification performance with multimodal features is superior to the one with unimodal features in the genre classification.

Video abstraction and summarization, being request in several applications, has address a number of researches to automatic video analysis techniques. The processes for automatic video analysis are based on the recognition of short sequences of contiguous frames that describe the same scene, shots, and key frames representing the salient content of the shot. Since effective shot boundary detection techniques exist in the literature, in this paper we will focus our attention on key frames extraction techniques to identify the low level visual features of the frames that better represent the shot content. To evaluate the features performance, key frame automatically extracted using these features, are compared to human operator video annotations.

In this paper, we present our work for automatic generation of textual metadata based on visual content analysis of video news. We present two methods for semantic object detection and recognition from a cross modal image-text thesaurus. These thesaurus represent a supervised association between models and semantic labels. This paper is concerned with two semantic objects: faces and Tv logos. In the first part, we present our work for efficient face detection and recogniton with automatic name generation. This method allows us also to suggest the textual annotation of shots close-up estimation. On the other hand, we were interested to automatically detect and recognize different Tv logos present on incoming different news from different Tv Channels. This work was done jointly with the French Tv Channel TF1 within the "MediaWorks" project that consists on an hybrid text-image indexing and retrieval plateform for video news.

To support more effective video retrieval at semantic level, we introduce a novel framework to achieve semantic video classification. This novel framework includes: (a) A semantic-senstive video content representation framework via principal video shots to enhance the quality of features (i.e., the ability of the selected low-level multimodal perceptual features to discriminate among various semantic video concepts); (b) A semantic video concept interpretation framework via flexible mixture model to bridge the semantic gap between the semantic video concepts and the low-level multimodal perceptual features; (c) A novel concept learning technique to integrate unlabeled samples with labeled samples for more accurate classifier training. Experimental results on semantic medical video classification are also presented to evaluate the performance of the proposed framework.

The long-term goals of the recently started Biomedia project at SFSU are to provide multimedia information systems and applications for the research and education needs of several projects in the SFSU Biology Department. These applications involve a considerable amount of images and image sequence data, in addition to traditional text, genomic, and experimental measurement data. Our systems will allow biology researchers and students to store, index, annotate, search, visualize, analyze, collaborate, and share a large amount of heterogeneous biomedical data. Our initial focus in BioMedia is the creation of collaborative WWW site for the Hedgehog gene pathway. The Hedgehog (Hh) protein super family constitutes a group of closely related secreted proteins that control many crucial processes during the embryogenesis of tissues. The overall goals of the Hh WWW Site project are as follows: a) to provide a WWW site to be used by researchers and students studying the Hedgehog gene pathway and made available to broad community, and b) to provide advanced and innovative functionality enabling easy usage and management, community based content submission and updates, and asynchronous collaboration between researchers and students. In this paper we present the status and first results in building and researching technologies necessary for this WWW site.

A model for evaluating the effectiveness of automatic recording of television programs by digital personal video recorders (PVRs) is presented. The model is used to evaluate the tradeoff between manual management of recording programs to a PVR archive and automatic management of recording. We show that a tradeoff exists between manual management in which the utility of a program is perfectly known but user awareness of available programs is limited versus automatic management in which utility estimates contain error but awareness is perfect. Experiments with the model show that the shape of this tradeoff is most governed by the shape of the distribution of user utility. As the percentage of programs with high user utility decreases relative to the average, the more effective automatic recording is likely to be despite errors in utility estimation. The shape of this tradeoff, however, is highly inelastic. Thus, improving utility estimates will not make automatic recording more effective if user awareness is sufficiently high.

In this paper we investigate the problem of integrating data sources with compex data types and complex data algebras, of the kind that is common in multimedia applications. We present the principles of mediation, and discuss why they are suitable for multimedia system. Then, we describe the mediation system Metropolis, and the specific solutions used for it that make it extendable to multimedia. Finally, we discuss one of the extensions currently under study: the integration of homologous abstract data types at the algebra level.

In this paper, we consider the problem of searching large image databases on the basis of the similarity between a query image submitted to the system and a set of target images in a database. In particular, we propose a system that combines results from several independent query configurations run in parallel. We provide a comparative analysis of performance of the individual configurations, which include four combinations of two color spaces (RGB and YIQ) and two wavelet composition algorithms (Haar and Daubechies), and an approach that combines the four configurations (fusion).

In this paper, we propose a home video summarization by representing a group of Representative frames (R-frames). The number of R-frames depends on shot characteristics which is shot duration and shot motion activity. For each shot, we apply an adaptive sub-sampling algorithm to extract the R-frames that contain only the high frame difference. When events of shots are not related to each other (One tape contains many events), it is desired to retrieve more information from more shots. Our algorithm allows users to select the number of shots appearing in the summarized video which give an optional way to understand the original sequence. In our experiments, we summarize video into variable number of shot appearing in the summary and evaluate the summary by users's subjective evaluations. We also summarize the home video by user's feedback. The user is asked to select the extracted R-frames as a training data. We apply the Support Vector Machine algorithm (SVM) to train and classify. The result from user's feedback shows that SVM retrieves the frames according to the user's desired.

A skimming system for movie content exploration is proposed using story units extracted via general tempo analysis of audio and visual data. Quite a few schemes have been proposed to segment video data into shots with low-level features, yet the grouping of shots into meaningful units, called story units here, is important and challenging. In this work, we detect similar shots using key frames and include these similar shots as a node in the scene transition graph. Then, an importance measure is calculated based on the total length of each node. Finally, we select sinks and shots according to this measure. Based on these semantic shots, a meaningful skims can be successfully generated. Simulation results will be presented to show that the proposed video skimming scheme can preserve the essential and significant content of the original video data.

In this paper we present a system for the localisation and tracking of billboards in streamed soccer matches. The application area for this research is the delivery of customised content to end users. When international soccer matches are broadcast, the diversity of the audience is very large and advertisers would like to be able to adapt the billboards to the different audiences. By replacing the billboards in the video stream this can be achieved. In order to build a more robust system, photometric invariant features are used. These colour features are less susceptible to the changes in illumination. Sensor noise is dealt with through variable kernel density estimation.

Many TV broadcasters and film archives are planning to make their collections available on the Web. However, a major problem with large film archives is the fact that it is difficult to search the content visually. A video summary is a sequence of video clips extracted from a longer video. Much shorter than the original, the summary preserves its essential messages. Hence, video summaries may speed up the search significantly. Videos that have full horizontal and vertical resolution will usually not be accepted on the Web, since the bandwidth required to transfer the video is generally very high. If the resolution of a video is reduced in an intelligent way, its content can still be understood. We introduce a new algorithm that reduces the resolution while preserving as much of the semantics as possible. In the MoCA (movie content analysis) project at the University of Mannheim we developed the video summarization component and tested it on a large collection of films. In this paper we discuss the particular challenges which the reduction of the video length poses, and report empirical results from the use of our summarization tool.

In this paper, dimension-reduced, decorrelated spectral features for general sound recognition are applied to segment conversational speech of both broadcast news audio and panel discussion television programs. Without a priori information about number of speakers, the audio stream is segmented by a hybrid metric-based and model-based segmentation algorithm. For the measure of the performance we compare the segmentation results of the hybrid method versus metric-based segmentation with both the MPEG-7 standardized features and Mel-scale Frequency Cepstrum Coefficients (MFCC). Results show that the MFCC features yield better performance compared to MPEG-7 features. The hybrid approach significantly outperforms direct metric based segmentation.

We present a neural network based approach to key frame extraction in the compressed domain. The proposed method is an amalgamation of both the MPEG-7 descriptors namely motion intensity descriptor and spatial activity descriptor. Shot boundary detection and block motion estimation techniques are employed prior to the extraction of the descriptors. The motion intensity (“pace of action”) is obtained using a fuzzy system that classifies the motion intensity into five categories proportional to the intensity. The spatial activity matrix determines the spatial distribution of activity (“active regions”) in a frame. A neural network is used to pick those frames as key frames which have high intensity and maximum spatial activity at the center of the frame. Results are compared against two well-known key frame extraction techniques to demonstrate the advantage and robustness of the proposed approach. Results show that the neural network approach performs much better than selecting first frame of the shot as a key frame and selecting middle frame of the shot as a key frame methods.

An object-oriented methodology is proposed to harmonize several different markup languages in this research. First, we adopt the Unified Modelling Language (UML) as the data model to formalize the concept and the process of the harmonization process between the eXtensible Markup Language (XML) applications. Then, we design the Harmonization eXtensible Markup Language (HXML) based on the data model and formalize the transformation between the Document Type Definitions (DTDs) of the original XML applications and HXML. The transformation between instances is also discussed. We use the harmonization of SMIL and X3D as an example to demonstrate the proposed methodology. This methodology can be generalized to various application domains.

Content-based video indexing and retrieval exhibits many challenging problems. Methods developed for processing of video for content search can be roughly categorized into temporal segmentation, spatial segmentation, and spatio-temporal video segmentation. The temporal segmentation aims to divide video into clips or (usually) camera shots. The spatial segmentation, on the other hand, seeks for the ROI’s (regions of interests) in video frames. The spatio-temporal segmentation is, however, more general since the video data is essentially spatial and temporal domain function in nature. It is expected that spatiotemporal methods describe the video content more completely and accurately as compared to the temporal and spatial segmentation of video independently. In the presented work, we approach the video-indexing problem by means of the spatio-temporal Radon projections. Specific projections are chosen for indexing the extracted features of a video clip. Pattern matching and pattern search are also studied via these projection-based features. Experiments show promising results for our approach.

Standardised image databases or rather the lack of them are one of the main weaknesses in the field of content based image retrieval (CBIR). Authors often use their own images or do not specify the source of their datasets. Naturally this makes comparison of results somewhat difficult. While a first approach towards a common colour image set has been taken by the MPEG 7 committee their database does not cater for all strands of research in the CBIR community. In particular as the MPEG-7 images only exist in compressed form it does not allow for an objective evaluation of image retrieval algorithms that operate in the compressed domain or to judge the influence image compression has on the performance of CBIR algorithms. In this paper we introduce a new dataset, UCID (pronounced "use it") - an Uncompressed Colour Image Dataset which tries to bridge this gap. The UCID dataset currently consists of 1338 uncompressed images together with a ground truth of a series of query images with corresponding models that an ideal CBIR algorithm would retrieve. While its initial intention was to provide a dataset for the evaluation of compressed domain algorithms, the UCID database also represents a good benchmark set for the evaluation of any kind of CBIR method as well as an image set that can be used to evaluate image compression and colour quantisation algorithms.

A novel CBIR system is described that incorporates both high level (semantic) and low level visual content. It is suitable for medical image information systems, to assist clinical diagnosis or for clinician training purposes. The system is XML-compliant and utilises MPEG-7 content descriptions and encoded metadata. The image retrieval process is driven by relevance feedback, enabling the indirect transferal of expert knowledge. A novel attribute visualisation facility enables the user to understand how the search criteria are modified and the effectiveness of the guidance provided. The relevance feedback visualisation can be used also to re-sort retrieved results according to the user's requirements and permit the interactive investigation of pertinent features. The effectiveness of the system is demonstrated by two examples from the field of dermatology. Evaluations show that combining the attribute visualisation with conventional retrieval techniques both increases user confidence levels and provides additional system functionality.

Medical images are at the heart of the healthcare diagnostic procedures. They have provided not only a noninvasive mean to view anatomical cross-sections of internal organs but also a mean for physicians to evaluate the patient’s diagnosis and monitor the effects of the treatment. For a Medical Center, the emphasis may shift from the generation of image to post processing and data management since the medical staff may generate even more processed images and other data from the original image after various analyses and post processing. A medical image data repository for health care information system is becoming a critical need. This data repository would contain comprehensive patient records, including information such as clinical data and related diagnostic images, and post-processed images. Due to the large volume and complexity of the data as well as the diversified user access requirements, the implementation of the medical image archive system will be a complex and challenging task. This paper discusses content standards for medical image metadata. In addition it also focuses on the image metadata content evaluation and metadata quality management.

Because video sequences consist of dynamic video objects in nature, video object motion is an effective feature in describing the content of video sequences. In this paper, we propose a method that converts motion vectors (MVs) on MPEG coded domain to a uniform set, independent of the frame type and the direction of prediction, and directly utilizes these normalized MVs (N-MVs) in understanding video contents. To obtain such uniform MV set, we proposed a new motion analysis method using Bi-directional Prediction-Independent Framework. Generally, it is impossible to directly compare an I frame without MV to other frame types such as B or P frame. But, our approach enables a frame-type independent representation that normalizes temporal features including frame type, MB encoding and MVs. In the experiments, we show that the proposed method is better than the conventional one in terms of performance.

With the development of computer technology, multimedia databases, especially image databases have been widely applied in many fields, among which medical image management is one of the most important. Traditional DBMS is good at managing formatted data. However, image is more complex than traditional data types both in structure and operation, which poses a big challenge to its wide application, so most current systems can only provide some basic storage and retrieval operations but lack the ability to meet such advanced management requirements as content query. Thus, content-Based image retrieval (CBIR) draws more and more research attention in the last few years. In CBIR, most research efforts are devoted into static image retrieval, but little has been done on evolution query of image sequences, which has great significance in medical image management. The formation and evolution of each kind of disease always demonstrate some specific patterns, which, if can be summarized, will help doctors especially those inexperienced ones further their understanding about the disease and make a better treatment plan. For example, when a new disease arises, doctors may hope to query the most similar disease evolutions from history records in the database based on the evolution of this new disease. Also, doctors can specify an abnormal image sequence manually as an example. Even, doctors may want to query those image sequences that exhibit the evolution of a certain disease. Most existing systems aim to assist doctors to diagnose a certain kind of disease, which usually have some drawbacks in their function of semantic retrieval. Many systems are a little difficult for the users to operate, or the functions are not sufficient. And the ability to support evolution query is even less. In this paper, we offer an evolution query solution based on medical image content. We design a system to implement our ideas. It consists of four modules, which are data process, feature management, model management and image retrieval. In addition to query formatted data and a single medical image by its content, the system can also support evolution query. This kind of query is to search image sequences evolving with time that can help doctors find out similar diagnose records from database as a reference. Image data and the related formatted data will first be preprocessed to eliminate erroneous and contradict information, and also to make up the missing information. And then feature extraction algorithms and object rule model are employed in object (or disease symptom such as lesion and hemorrhage, etc) recognition. Finally, the system will analyze the semantics of each medical image with static semantic model, such as to determine the kind and the phase of the disease. On that basis, evolution query at both object and semantic levels can be provided. This paper mainly elaborates on evolution query methods on medical image databases. Different from querying a single image, querying disease evolution concerns with the comparison of two image sequences evolving with time. It needs to search a certain number of similar image sequences given a target one. Just as it is not easy to define the similarity of two single images, it seems far more difficult to define the similarity of two disease evolutions. The reason is that there are too more factors that can be compared between two evolutions and even doctors themselves can not tell whether they are similar or not generally. However, in most cases, the comparison of only one factor between two evolutions looks more meaningful in practice because the similarity of multiple factors can be calculated by adding weights to each individual factor. So, this paper first discusses the factors that can be compared between two image sequences, or the selection of feature values. Each image can be represented by a feature vector formed by a number of feature values. Image evolution is the result of feature value evolutions. But feature selection is dependent on specific application fields. Doctors care for different feature values in different diseases. We can only give some possibly common feature values as an example. Typical feature values can consists of the number of symptom types, the number, the average location and size of the symptoms. Thus the comparison of two evolutions turns into the comparison of two curves in graphic view by considering the evolution of each feature value individually. Because we want to compare the similarity of two evolutions but not some specific feature values, we have to normalize the curves to remove the influence of vertical offset. This paper proposes two methods of normalization, that is, to normalize the curve as a whole or to treat each part separately. The former can only remove part of vertical movement. So we use the latter instead to break the whole curve into a sequence of independent lines. This paper then gives many methods to calculate the similarity of two curves after normalization. To calculate the sum of angles formed by two lines in each time period is possible, but it seems a little complex in calculation. In addition, it is not very sensitive when the slope becomes large. The alternative is to calculate the sum of feature value differences, which uses the sum of line differences at each time point as a symbol for evolution similarity. This method is identical to calculate the sum of the area formed by two lines in each time period. We use the second method as the basis of our calculation. The less the difference is, the more similar the two curves are. But this method still has some drawbacks and needs to be improved little by little. Firstly, line difference in each time period does not reflect the difference of orientation, because two lines with both the same and opposite orientations can possibly have the same line difference. However, two lines both going up or both going down look more similar than those with one going up and the other going down. So our system enlarges line differences by multiplying a constant weight larger than one to discriminate this situation. Secondly, the sum of line differences treats each time period individually, thus does not take into account time continuity of the evolution. But in practice, two curves showing continuous similarity look more similar than those showing separate similarity. So we use a variant weight instead of a constant one in calculating line differences. If two curves show continuous similarity, we decrease the weight continuously to reduce line difference, which well addresses this issue. We provide two interfaces: query by example and query by concept. In the form of query by example, we allow users to specify an existing medical image sequence as an example. We can also allow users to describe an abnormal image sequence manually by providing some useful tools. Through personalized templates, we link certain image sequences with advanced concepts to support concept based semantic queries. Content based evolution retrieval of medical images is a brand new research field. Although it is very important in practice, little effort has been devoted to it by far. The contribution of this paper includes the following: we first analyze the great significance of evolution query on medical images in application. And then discuss the possible factors that can be compared between two evolutions, or the selection of feature values. Later, we propose two normalization methods to remove the influence of vertical offset. After that, we give several possible methods to calculate the similarity of two evolutions and elaborate on their advantages and the drawbacks. Based on these efforts, we offer an improved method and well address the issue of evolution comparison. In our future work, we will try to build an efficient index structure to achieve fast evolution query in medical image databases.

Traditional video coding uses fixed keyframe setting, which cannot exert all the potentials of inter-frame prediction. Dynamic adaptive keyframe setting can improve the efficiency and quality of video coding. However, it generally requires time-consuming preprocessing of video analysis, which prevents it from being practical. In this paper, we propose a novel method of keyframe setting: Fast Dynamic Adaptive Keyframe Setting. The method adopts a quasi single-pass strategy of exploiting information generated by original motion-estimation, so the additional computing costs are acceptable. The method can improve both objective and subjective quality, and can be applied to any motion-estimation based video coding. Experimental results on both MPEG-2 codec and H.264 codec prove that the method is viable and valuable.

Colonoscopy is an important screening procedure for colorectal cancer. During this procedure, the endoscopist visually inspects the colon. Human inspection, however, is not without error. We hypothesize that colonoscopy videos may contain additional valuable information missed by the endoscopist. Video segmentation is the first necessary step for the content-based video analysis and retrieval to provide efficient access to the important images and video segments from a large colonoscopy video database. Based on the unique characteristics of colonoscopy videos, we introduce a new scheme to detect and remove blurry frames, and segment the videos into shots based on the contents. Our experimental results show that the average precision and recall of the proposed scheme are over 90% for the detection of non-blurry images. The proposed method of blurry frame detection and shot segmentation is extensible to the videos captured from other endoscopic procedures such as upper gastrointestinal endoscopy, enteroscopy, cystoscopy, and laparoscopy.

In molecular biology, retrieval of 3D protein structures is a very important topic. In this paper. From the view of geometry, we propose three types of characteristics (e.g. VDP,DDP,ADP) for describing 3D protein structures in order to perform similarity search and classification. The goal of the approach is to reduce the 3D protein structures matching problem to the comparison of the probability distribution. Experimental results show that our method is robust and invariant with respect to translation, rotation.

Motion track is an important feature to show the spatio-temporal relationship of a video object in a video. In this paper, we propose a novel motion track representation based on MPEG-7 motion descriptor. A new descriptor is proposed to represent the motion track in the X-Y plane and the trend of velocity changes. Moreover, a new similarity measure for comparing two motion tracks based on the motion trajectory and velocity differences is proposed. The trajectory is compared by the properties of the polynomials, and the velocity is compared by the different trends. Furthermore, the motion track segmentation method is proposed to handle a complicated motion behavior and the relevance feedback is used to improve the query results. Experiment results show that this approach has a higher precision than existing approaches.

This paper presents a new approach to content-based image retrieval by using dynamic indexing and guided search in a hierarchical structure, and extending data mining and data warehousing techniques. The proposed algorithms include: a wavelet-based scheme for multiple image feature extraction, the extension of a conventional data warehouse and an image database to an image data warehouse for dynamic image indexing, an image data schema for hierarchical image representation and dynamic image indexing, a statistically based feature selection scheme to achieve flexible similarity measures, and a feature component code to facilitate query processing and guide the search for the best matching. A series of case studies are reported, which include a wavelet-based image color hierarchy, classification of satellite images, tropical cyclone pattern recognition, and personal identification using multi-level palmprint and face features.

In this paper, we design and implement a concept-based image retrieval system using feature information, more specifically, edge histogram description. The general edge histogram framework is a novel index mechanism which allows us to describe a content of images. However, there is a significant drawback in the framework that it can not accommodate a concept-based retrieval. Even if images are only conceptually related with user queries, it may be capable of proving them to be irrelevant since their features can be different each other. Our system adapts an edge histogram descriptor and includes a knowledge used for capturing concepts from images. In the knowledge base, a concept is expressed as some of templates, which can be described by common edge histograms for the images to represent the concept well. The templates can be generated by clustering the training images related with a concept. Consequently, since an image can also be matched with some of the templates, our system is able to support an automatic mechanism for indexing the image with the concept. The indexing mechanism enables users to retrieve the images related with a query which is formulated with their intended concepts. In addition, we also demonstrate that our concept-based approach makes a favorable comparison with an approach based on color or edge features.