This PDF file contains the front matter associated with SPIE Proceedings Volume 9473, including the Title Page, Copyright information, Table of Contents, Introduction (if any), and Conference Committee listing.

A vast amount of temporal information are collected and stored which leads to a need for techniques to extract useful high-level information from the data, such as recognizable activities or events. This paper proposes a framework for streaming analysis of time series data, in this case track data from video sequences, which can recognize events without supervision and memorize them by building temporal contexts. The memorized historical data are then used to predict the future and detect anomalous activities or events. An incremental clustering method is used to recognize and learn the event without training. A memorization method of double localization, including relative and absolute localization, is proposed to model the temporal context. The first- order Markov chain is used as an example to relatively localize an event in the temporal space, while a temporal map is used to absolutely localize an event. By setting proper coordinates for the temporal map, prior temporal patterns can be included before building more delicate models to find subtle patterns. Finally, the predictive model is built based on the method of memorization. The "Edinburgh Informatics Forum Pedestrian Data Set", which offers about 1000 observed trajectories of pedestrians detected in camera images each working day for several months, is used as an example to illustrate the framework. The trajectories are incrementally recognized as activities, and the activities are further used to describe the situation (event) in the scene. By using the proposed framework, we not only get current high level information of the scene, such as activities and events that occur in the scene, but we can also reason whether current activities or events are "normal" by referring to the temporal context learned from historical data. For the data considered here, a total of 92,000+ observed trajectories and their temporal patterns over one year can be summarized by a temporal map of 16 events and a probability (transition) matrix between them. Our proposed framework offers an effective method of extracting useful and manageable information from a huge amount of raw trajectory data.

Change detection between two images in the presence of degradations is an important problem in the computer vision community, more so for the aerial scenario which is particularly challenging. Cameras mounted on moving platforms such as aircrafts or drones are subject to general six-dimensional motion as the motion is not restricted to a single plane. With CMOS cameras increasingly in vogue due to their low power consumption, the inevitability of rolling-shutter (RS) effect adds to the challenge. This is caused by sequential exposure of rows in CMOS cameras unlike conventional global shutter cameras where all pixels are exposed simultaneously. The RS effect is particularly pronounced in aerial imaging since each row of the imaging sensor is likely to experience a different motion. For fast-moving platforms, the problem is further compounded since the rows are also affected by motion blur. Moreover, since the two images are shot at different times, illumination differences are common. In this paper, we propose a unified computational framework that elegantly exploits the scarcity constraint to deal with the problem of change detection in images degraded by RS effect, motion blur as well as non-global illumination differences. We formulate an optimization problem where each row of the distorted image is approximated as a weighted sum of the corresponding rows in warped versions of the reference image due to camera motion within the exposure period to account for geometric as well as photometric differences. The method has been validated on both synthetic and real data.

Existing image registration methods that work in the presence of motion blur assume the scene to be fronto-parallel. In this work, we extend the state-of-the-art by cueing motion blur itself to infer plane inclination. This is achieved by matching extremities of blur kernels computed at different locations in the image. Because these extremities correspond to the same homography, we show that it is possible to find the orientation. Following this, we propose a registration method that reorients the source image to the target plane within a reblur-difference framework to detect the actual changes.

Aerial video surveillance has advanced significantly in recent years, as inexpensive high-quality video cameras and airborne platforms are becoming more readily available. Video has become an indispensable part of military operations and is now becoming increasingly valuable in the civil and paramilitary sectors. Such surveillance capabilities are useful for battlefield intelligence and reconnaissance as well as monitoring major events, border control and critical infrastructure. However, monitoring this growing flood of video data requires significant effort from increasingly large numbers of video analysts. We have developed a suite of aerial video exploitation tools that can alleviate mundane monitoring from the analysts, by detecting and alerting objects and activities that require analysts’ attention. These tools can be used for both tactical applications and post-mission analytics so that the video data can be exploited more efficiently and timely. A feature-based approach and a pixel-based approach have been developed for Video Moving Target Indicator (VMTI) to detect moving objects at real-time in aerial video. Such moving objects can then be classified by a person detector algorithm which was trained with representative aerial data. We have also developed an activity detection tool that can detect activities of interests in aerial video, such as person-vehicle interaction. We have implemented a flexible framework so that new processing modules can be added easily. The Graphical User Interface (GUI) allows the user to configure the processing pipeline at run-time to evaluate different algorithms and parameters. Promising experimental results have been obtained using these tools and an evaluation has been carried out to characterize their performance.

The Motion Imagery Standards Board (MISB) has created the Video National Imagery Interpretability Rating Scale (VNIIRS). VNIIRS extends NIIRS to scene characterization from streaming video to include object recognition of various targets for a given size. To apply VNIIRs for target tracking, there is a need to understand the operating conditions of the sensor type, environmental phenomenon, and target behavior (SET). In this paper, we explore VNIIRS for target tracking given the sensor resolution to support the relative tracking performance using track success. The relative assessment can be used in relation to the absolute target size associated with the VNIIRS. In a notional analysis, we determine the issues and capabilities of using VNIIRS video quality ratings to determine track success. The outcome of the trade study is an experiment to understand how to use VNIIRS can support target tracking evaluation.

This paper presents AESOP, a software tool for automatic event detection in video. AESOP employs a super- vised learning approach for constructing event models, given training examples from different event classes. A trajectory-based formulation is used for modeling events with an aim towards incorporating invariance to changes in the camera location and orientation parameters. The proposed formulation is designed to accommodate events that involve interactions between two or more entities over an extended period of time. AESOP's event models are formulated as HMMs to improve the event detection algorithm's robustness to noise in input data and to achieve computationally efficient algorithms for event model training and event detection. AESOP's performance is demonstrated on a wide range of different scenarios, including stationary camera surveillance and aerial video footage captured in land and maritime environments.

Tracking individuals in areas such as dense urban environments and building interiors is desirable for numerous critical applications, but has been problematic mainly because of the unreliability or unavailability of GPS in many locations of interest. To date, tracking applications that utilize inertial sensors within smart devices have had varied degrees of success: accuracy typically dips below that of standard GPS within minutes and depends strongly on the quality of the sensors in the device, as well as the location that the device is carried on the body. In this paper we present a sensor module that interfaces with modern smart devices and which utilizes a low-cost, commercial-off-the-shelf, 9-axis IMU and pressure sensor to provide an advanced pedestrian dead reckoning solution. The sensor module is designed to communicate with the smart device (e.g., iOS, Android or Windows) via the audio jack and is intended for use as a beltmounted pedestrian tracker. In addition to describing the device hardware and functionality, we present our approach to processing the sensor module data streams to determine a user’s position. Results using the prototype sensor module in operationally relevant scenarios is presented and discussed.

As technology and internet use grows at an exponential rate, video and imagery data is becoming increasingly important. Various techniques such as Wide Area Motion imagery (WAMI), Full Motion Video (FMV), and Hyperspectral Imaging (HSI) are used to collect motion data and extract relevant information. Detecting and identifying a particular object in imagery data is an important step in understanding visual imagery, such as content-based image retrieval (CBIR). Imagery data is segmented and automatically analyzed and stored in dynamic and robust database. In our system, we seek utilize image fusion methods which require quality metrics. Many Image Fusion (IF) algorithms have been proposed based on different, but only a few metrics, used to evaluate the performance of these algorithms. In this paper, we seek a robust, objective metric to evaluate the performance of IF algorithms which compares the outcome of a given algorithm to ground truth and reports several types of errors. Given the ground truth of a motion imagery data, it will compute detection failure, false alarm, precision and recall metrics, background and foreground regions statistics, as well as split and merge of foreground regions. Using the Structural Similarity Index (SSIM), Mutual Information (MI), and entropy metrics; experimental results demonstrate the effectiveness of the proposed methodology for object detection, activity exploitation, and CBIR.

A method for generating and utilizing structure from motion (SfM) uncertainty estimates within image-based pose estimation is presented. The method is applied to a class of problems in which SfM algorithms are utilized to form a geo-registered reference model of a particular ground area using imagery gathered during flight by a small unmanned aircraft. The model is then used to form camera pose estimates in near real-time from imagery gathered later. The resulting pose estimates can be utilized by any of the other onboard systems (e.g. as a replacement for GPS data) or downstream exploitation systems, e.g., image-based object trackers. However, many of the consumers of pose estimates require an assessment of the pose accuracy. The method for generating the accuracy assessment is presented. First, the uncertainty in the reference model is estimated. Bundle Adjustment (BA) is utilized for model generation. While the high-level approach for generating a covariance matrix of the BA parameters is straightforward, typical computing hardware is not able to support the required operations due to the scale of the optimization problem within BA. Therefore, a series of sparse matrix operations is utilized to form an exact covariance matrix for only the parameters that are needed at a particular moment. Once the uncertainty in the model has been determined, it is used to augment Perspective-n-Point pose estimation algorithms to improve the pose accuracy and to estimate the resulting pose uncertainty. The implementation of the described method is presented along with results including results gathered from flight test data.

In order to better understand the issues associated with Full Motion Video (FMV) geopositioning and to develop corresponding strategies and algorithms, an integrated test bed is required. It is used to evaluate the performance of various candidate algorithms associated with registration of the video frames and subsequent geopositioning using the registered frames. Major issues include reliable error propagation or predicted solution accuracy, optimal vs. suboptimal vs. divergent solutions, robust processing in the presence of poor or non-existent a priori estimates of sensor metadata, difficulty in the measurement of tie points between adjacent frames, poor imaging geometry including small field-of-view and little vertical relief, and no control (points). The test bed modules must be integrated with appropriate data flows between them. The test bed must also ingest/generate real and simulated data and support evaluation of corresponding performance based on module-internal metrics as well as comparisons to real or simulated “ground truth”. Selection of the appropriate modules and algorithms must be both operator specifiable and specifiable as automatic. An FMV test bed has been developed and continues to be improved with the above characteristics. The paper describes its overall design as well as key underlying algorithms, including a recent update to “A matrix” generation, which allows for the computation of arbitrary inter-frame error cross-covariance matrices associated with Kalman filter (KF) registration in the presence of dynamic state vector definition, necessary for rigorous error propagation when the contents/definition of the KF state vector changes due to added/dropped tie points. Performance of a tested scenario is also presented.

Geostatistical modeling of spatial uncertainty has its roots in the mining, water and oil reservoir exploration communities, and has great potential for broader applications as proposed in this paper. This paper describes the underlying statistical models and their use in both the estimation of quantities of interest and the Monte-Carlo simulation of their uncertainty or errors, including their variance or expected magnitude and their spatial correlations or inter-relationships. These quantities can include 2D or 3D terrain locations, feature vertex locations, or any specified attributes whose statistical properties vary spatially. The simulation of spatial uncertainty or errors is a practical and powerful tool for understanding the effects of error propagation in complex systems. This paper describes various simulation techniques and trades-off their generality with complexity and speed. One technique recently proposed by the authors, Fast Sequential Simulation, has the ability to simulate tens of millions of errors with specifiable variance and spatial correlations in a few seconds on a lap-top computer. This ability allows for the timely evaluation of resultant output errors or the performance of a “down-stream” module or application. It also allows for near-real time evaluation when such a simulation capability is built into the application itself.

Recent advances in distributed databases and computing have transformed the landscape of spatio-temporal machine learning. This paper presents GeoMesa, a distributed spatio-temporal database built on top of Hadoop and column-family databases such as Accumulo and HBase, that includes a suite of tools for indexing, managing and analyzing both vector and raster data. The indexing techniques use space filling curves to map multi-dimensional data to the single lexicographic list managed by the underlying distributed database. In contrast to traditional non-distributed RDBMS, GeoMesa is capable of scaling horizontally by adding more resources at runtime; the index rebalances across the additional resources. In the raster domain, GeoMesa leverages Accumulo's server-side iterators and aggregators to perform raster interpolation and associative map algebra operations in parallel at query time. The paper concludes with two geo-time data fusion examples: using GeoMesa to aggregate Twitter data by keywords; and georegistration to drape full-motion video (FMV) over terrain.

The domain of Geospatial Intelligence Analysis is rapidly shifting toward a new paradigm of Activity Based Intelligence (ABI) and information-based Tipping and Cueing. General requirements for an advanced ABIAA system present significant challenges in architectural design, computing resources, data volumes, workflow efficiency, data mining and analysis algorithms, and database structures. These sophisticated ABI software systems must include advanced algorithms that automatically flag activities of interest in less time and within larger data volumes than can be processed by human analysts. In doing this, they must also maintain the geospatial accuracy necessary for cross-correlation of multi-intelligence data sources.

Historically, serial architectural workflows have been employed in ABIAA system design for tasking, collection, processing, exploitation, and dissemination. These simpler architectures may produce implementations that solve short term requirements; however, they have serious limitations that preclude them from being used effectively in an automated ABIAA system with multiple data sources.

This paper discusses modern ABIAA architectural considerations providing an overview of an advanced ABIAA system and comparisons to legacy systems. It concludes with a recommended strategy and incremental approach to the research, development, and construction of a fully automated ABIAA system.

Georeferenced data of various modalities are increasingly available for intelligence and commercial use, however effectively exploiting these sources demands a unified data space capable of capturing the unique contribution of each input. This work presents a suite of software tools for representing geospatial vector data and overhead imagery in a shared high-dimension vector or embedding" space that supports fused learning and similarity search across dissimilar modalities. While the approach is suitable for fusing arbitrary input types, including free text, the present work exploits the obvious but computationally difficult relationship between GIS and overhead imagery. GIS is comprised of temporally-smoothed but information-limited content of a GIS, while overhead imagery provides an information-rich but temporally-limited perspective. This processing framework includes some important extensions of concepts in literature but, more critically, presents a means to accomplish them as a unified framework at scale on commodity cloud architectures.

Flares are used as valuable electronic warfare assets for the battle against infrared guided missiles. The trajectory of the flare is one of the most important factors that determine the effectiveness of the counter measure. Reconstruction of the three dimensional (3D) position of a point, which is seen by multiple cameras, is a common problem. Camera placement, camera calibration, corresponding pixel determination in between the images of different cameras and also the triangulation algorithm affect the performance of 3D position estimation. In this paper, we specifically investigate the effects of camera placement on the flare trajectory estimation performance by simulations. Firstly, 3D trajectory of a flare and also the aircraft, which dispenses the flare, are generated with simple motion models. Then, we place two virtual ideal pinhole camera models on different locations. Assuming the cameras are tracking the aircraft perfectly, the view vectors of the cameras are computed. Afterwards, using the view vector of each camera and also the 3D position of the flare, image plane coordinates of the flare on both cameras are computed using the field of view (FOV) values. To increase the fidelity of the simulation, we have used two sources of error. One is used to model the uncertainties in the determination of the camera view vectors, i.e. the orientations of the cameras are measured noisy. Second noise source is used to model the imperfections of the corresponding pixel determination of the flare in between the two cameras. Finally, 3D position of the flare is estimated using the corresponding pixel indices, view vector and also the FOV of the cameras by triangulation. All the processes mentioned so far are repeated for different relative camera placements so that the optimum estimation error performance is found for the given aircraft and are trajectories.

Accurately recognizing building roof style leads to a much more realistic 3D building modeling and rendering. In this paper, we propose a novel system for image based roof style classification using machine learning technique. Our system is capable of accurately recognizing four individual roof styles and a complex roof which is composed of multiple parts. We make several novel contributions in this paper. First, we propose an algorithm that segments a complex roof to parts which enable our system to recognize the entire roof based on recognition of each part. Second, to better characterize a roof image, we design a new feature extracted from a roof edge image. We demonstrate that this feature has much better performance compared to recognition results generated by Histogram of Oriented Gradient (HOG), Scale-invariant Feature Transform (SIFT) and Local Binary Patterns (LBP). Finally, to generate a classifier, we propose a learning scheme that trains the classifier using both synthetic and real roof images. Experiment results show that our classifier performs well on several test collections.

Intelligence, Surveillance, and Reconnaissance (ISR) missions involve complex analysis of sensor data that can benefit from the exploitation of geographically aligned context. In this paper we discuss our approach to utilizing geo-registered imagery and context for the purpose of aiding ISR detection and tracking applications. Specifically this includes rendering context masks on imagery, increasing the speed at which detection algorithms process data, providing a way to intelligently control detection density for given ground areas, identifying difficult traffic terrain, refining peak suppression for congested areas, reducing target center of mass location errors, and increasing track coverage and duration through track prediction error robustness.

The VideoPlus^®-Aware (VPA) system enables autonomous video-based target detection, tracking and classification. The system stabilizes video and operates completely autonomously. A statistical background model enables robust acquisition of moving targets, while stopped targets are tracked using feature-based detectors. An ensemble classifier is trained for automated detection and classification of dismounts (i.e., humans) and a planar scene model is used to both improve system performance and reduce false positives. A formal evaluation of the VPA system was performed by the government, to quantify the system’s abilities to detect, track, and classify, humans. The evaluation provided 811 separate data points gathered over a period of four days with an overall probability of sensing of 99.9%. The probability of detection was 86.2% and the percentage of correct action classification was 82%. The data provided a False Alarm Rate of 0 per hour and Nuisance Alarm Rate of 0.72 per hour. Dismounts were reliably classified with pixel heights as low as 25 pixels. Real-time automated detection, tracking, and classification of targets with low false positive rates was achieved, even with few pixels on target. The planar scene model based optimizations were sufficient to dramatically reduce the runtime of sliding-window classifiers.