This PDF file contains the front matter associated with SPIE Proceedings Volume 12621, including the Title Page, Copyright information, Table of Contents, and Conference Committee information.

Structural Health Monitoring (SHM) is critical to ensuring the safety of structures such as bridges, tunnels, and dams. Despite some sensors being highly accurate, it is not always feasible to interrupt the serviceability of the structure for data collection. Within this framework, remote sensing methods such as the Ground-Based Interferometric Synthetic Aperture Radar (GB-SAR) have shown their capability in remotely collecting data of multiple targets simultaneously with a high sampling rate. However, detecting targets in their exact position is currently an area that requires further investigation for this technology. The present research focuses on developing a field investigation methodology to limit uncertainties and raise awareness about the significance of data collected by GB-SAR aided by augmented reality (AR). To this effect, head-mounted and mobile AR can support the use of techniques, such as GB-SAR, which work on fundamental geometrical principles, by providing guidance markers in real time for positional and reference information. The integration of both technologies can allow to pre-visualise the optimal position for data collection by aiding to match the structural targets under investigation within the area of interest. The proposed methodology is here implemented in clinical laboratory conditions to investigate the sensor’ sensitivity against testing parameters, such as the radar position and the distance to targets. The proposed methodology will contribute to collecting data with a higher accuracy and a lower uncertainty compared to other non-destructive methods utilised in the field. This study demonstrates the potential of using AR to enhance remote sensing methods for SHM and it builds up the foundation for future development into a more comprehensive SHM approach.

Transport assets are progressively more exposed to several issues, including climate change, and vulnerability to natural hazards such as subsidence, landslides, and seismic phenomena, to mention a few. They can affect the structural integrity of infrastructures causing damages and deteriorations. They support the prioritization of the most effective countermeasures. We propose a method based on Data-Fusion approach, merging multi-source and multi-scale information acquired by multi-sensors systems. It enhances the interpretation of fused data under a holistic framework. For this purpose, processing methods of spaceborne data, such as Multi-temporal SAR Interferometry, are used to detect displacements of transport assets, with millimetre accuracy. This technique is complemented by more detailed data detected by UAVs and Ground Based Non-Destructive Testing Methods, including laser scanner surveys and by prospecting data collected by Ground Penetrating Radar. All these data are simultaneously fused and digitally integrated into a comprehensive and high-resolution Digital Twin, providing a useful tool to support operators and public authorities to identify and prioritize maintenance and rehabilitation actions. According to this holistic framework, the introduction of the Digital Twin, which replicates a real asset in a virtual or augmented reality environment, represents a relevant and strategic tool or facility to tackle effectively and efficiently the maintenance issues, enhancing resilience and operating life of the infrastructures. This study shows an experimental application of the proposed methodology by fusing spatial, aerial, ground-based, and geophysical surveys using also augmented reality tools, to have the best interface for visualizing the twin in the real environment. This allows smart evaluation and monitoring which has resoluted, efficient, holistic and up-to-date information.

Building Information Modeling (BIM) is a software-based parametric design approach that enables complete interoperability between all of the various actors engaged in a design or management process. Notwithstanding it was particularly developed for construction projects, it has since been applied to a variety of fields, including the design of transportation infrastructures and, more recently, cultural heritage. It has primarily been used in this area to improve the accuracy and efficiency of activities aimed at stabilizing and restoring historical structures. The purpose of the current study is to show how BIM is implemented by multisensors date and how the use of BIM can significantly raise the quality of virtual reconstructions and digital valorizations of historical structures, particularly when the rate of conservation is low. In spite of the fact that modern digital reconstruction models are frequently examined from an archaeological standpoint, their structural coherence has never been examined. This implies that many virtual models are likely to depict historically reliable structures that, due to their lack of structural logic, would not support their weight according to their geometric features and construction methods/materials. The research suggests a novel BIM-based methodology capable of both guiding and structurally testing archaeological reconstruction theories. The following procedure can serve as a broad representation of the model: 1- Multisensors survey of the emerging: gathering information from cursory historical investigations (topographic data, laser scanner, aero photogrammetry, satellite images); 2- Multisensors survey of the hidden: gathering information from ground-penetrating radar, electrical tomography, and magnetometry hypogeal scans; 3- Mechanical characterization: collecting data on the materials used in the discovery and confirming their mechanical properties through load stress tests; 4- Virtual reconstruction: the formulation of a potential hypothesis related to structural and morphological characteristics that are known to have existed during the referred historical eras; 5- Structural test: engineering and structural verification of the proposed theory using finite element algorithms and graphic statics techniques. In the framework of the Project BIMHERIT, funded by Regione Lazio, the proposed methodology was tested at a preliminary stage on the archaeological site of the Villa and Circus of Maxentius along the Ancient Appian Way in Rome. All planned activities have been shared with and authorized by the Sovrintendenza Capitolina ai Beni Culturali (DTC Lazio Call, Prot. 305-2020-35609).

Hot boxes, which refer to overheated rail-road car wheels and bearings, pose a significant threat to railway operations. Failure to detect and address hot boxes promptly can lead to catastrophic accidents such as derailments and fires. Current way-side hot box detectors operate on the principle that an axle bearing will emit a large amount of heat when it is close to failing. They require principally an infrared (IR) sensor mounted at specific locations along the track, and a signal source coming from a wayside detectors or track circuits to detect if a train is approaching. The IR sensors scanning location, however, should be carefully selected to avoid under/over predicting the operating temperature of the axle bearings and wheels. The dependency of a signal source to activate the system may be problematic as well, not to mention its implementation and maintenance costs. The main contribution of this paper lies with the development of an automatic hot box detection, tracking and counting method by only using the IR cameras. The method combines the YOLO algorithm with the Kalman filter as a tracker. The method was tested with original datasets built with IR images taken from two wayside camera models, cooled and uncooled cameras. The experiments have been conducted on both freight and passenger trains at different times of the day, under clear weather conditions. Apart from the promising results obtained by YOLO, it is found that the Kalman filter further improves the tracking and thus the detection performance, minimizing thereby the incorrect detection or missed detection.

The timely detection of leakage in water mains is an issue that is relevant to the sustainable and efficient use of natural resources and the prevention of environmental hazards and risks for citizens. Consequently, the development of non-destructive techniques capable of detecting and localizing water leaks in buried pipelines is of huge interest. In this contribution, we present an artificial intelligence tool to perform automatic leakage detection from ground penetrating radar tomographic images. Ground penetrating radar is a prominent technology for subsoil inspection based on the remote interaction of microwave signals with buried anomalies, but its results require expert-users and are prone to subjective interpretation. This can be counteracted by processing raw-data using microwave tomography algorithms, which are capable of delivering more easily interpretable images. However, tomographic images can be still very difficult to interpret when the assumptions underlying the algorithm fail and therefore do not lead to conclusive results. To overcome this issue, we cast the leakage-detection problem as an image segmentation task, in which the popular convolutional neural network U-NET is trained to turn tomographic images obtained from raw-data processing into binary images clearly depicting the location of the leaks. Preliminary results with full-wave synthetic data confirm the potential of the proposed approach.

Tree trunk monitoring is crucial for both industrial and environmental applications, as well as for ecological preservation. To date, Ground Penetrating Radar (GPR) has shown promising results in mapping the inner structures of tree trunks, which could lead to novel forest management practices and support preventing the collapse of trees in urban areas. This is crucial to avoid harming the lives and properties of citizens. Nevertheless, it has been observed that the irregular geometry of the tree trunk surface can cause mismatches in associating the GPR traces to their correct position along the bark and, hence, distortion in the processed GPR images. This can escalate into an incorrect location of internal targets of interest, e.g., holes and decays within the tree trunk. To this end, this research aims to correct the effect of GPR data distortion along complex tree trunk surface geometries, by locating the GPR A-scans (traces) in their actual position. First, Smartphone Light Detection and Ranging (LiDAR) technology, a multi-angle measuring ruler, and a high-resolution camera are employed to extract the irregular contours of a sample tree bark. GPR data are subsequently collected along the measured surface and correlated with the respective profile’s traces. The methods’ accuracy is investigated by way of comparison between the different employed edge detection techniques through laboratory measurements on an oak tree trunk sample. The results show that the Smartphone LiDAR technology improves the accuracy of the reconstructed images of tree trunks by providing high-precision and affordable coordinate information. This integrated technique has great potential for enhancing the prediction of the location of decays in tree trunks and creating more accurate GPRimage- based models. The achieved results pave the way for novel forest management practices and contribute to preserving ecological systems.

Automated systems increase their requirements in all fields, tightening their performance requirements in aspects like reliability, and ease of manipulation. Within this communication, we will present the development of a compact perception unit which includes a 3D lidar, RGB and thermal imaging for advanced perception purposes. The proposed unit intends to solve the usual hardware problems that software developers intend to solve in field applications. The basic features and performance of the system will be presented, and the applicability of the multimodal sensing approach presented to different applications in security, autonomous vehicles, and other application areas will be overviewed with examples.

Detecting defects that may arise in weldings used in either critical or non-critical industrial applications is an extensive area of active research. As such, non-destructive tests are often required, mainly to preserve the integrity of normal samples while discarding defective ones. Amongst these types of tests, the most relevant are visual inspections which, however, require a considerable effort by a domain expert. Still, as these tasks require relevant efforts and could be biased by subjectivity and inexperience, the development of automated, objective tools that provide early warnings about the occurrence of one or more anomalies has become essential. To deal with these issues, this work proposes a framework for detecting anomalies in linear aluminum welding. Specifically, the framework starts by acquiring the three-dimensional representation of the welding using a 3D laser profiler. Afterward, a semi-supervised approach is followed by training a deep autoencoder on non-defective data samples, allowing the model to learn a mapping function that successfully reconstructs non-defective weldings with a small reconstruction error while providing a large error for abnormal data samples. Hence, by applying a proper threshold on the reconstruction error, which can be estimated via statistical analysis, the framework can provide real-time early warnings concerning surface defects on the welding. Different structures of autoencoder have been tested and found to reach an F₁ score of over 90%.

Model predictive control (MPC), with prediction and control horizons under multivariable constraints, is an advanced form of model-based control and is commonly used to implement the path following of autonomous vehicles. Conventionally, MPC requires a kinematic or dynamic model of the vehicle to optimize the controller. When a nonlinear kinematic model is utilized, the model should be linearized prior to use by the MPC. However, the computational cost of model linearization is rather high, and hence implementing the MPC in real-time is extremely difficult. Furthermore, estimating the parameters of a classic dynamic model is difficult. Accordingly, the present study first uses a data-driven system identification (system ID) approach to estimate the dynamic model of the considered vehicle (a tracked unmanned ground vehicle (UGV)) as a state-space linear dynamic model. It is shown that the identified model with two-channel inputs and three-channel outputs achieves a fitting of more than 45% between the predicted and measured position and posture of the vehicle. The S-curve and L-shape path-following performance of the tracked vehicle based on MPC with the identified state-space dynamic model is significantly improved. Furthermore, a nonlinear MPC with a long short-term memory (LSTM) model is utilized to adapt different kinds of working environments such as on sand land or in rainy day. According to different system input and output, a suitable model via the LSTM network is estimated in real time and utilized in the nonlinear MPC to enforce the tracked vehicle to follow the path accurately.

Direct Energy Deposition (DED) is an additive manufacturing technology that allows for both the realization of complex shape objects and for repairing of damaged or worn metallic components. A high-power laser beam melts the substrate and the metallic powder particles ejected by nozzles converging to the same spot. The laser beam spans the substrate surface, creating overlapping solid tracks and thus building up parts in a layer-wise fashion. Despite fine parameters optimization, the process instability may lead to layer height variations that, accumulating over time, could affect the stand-off distance, i.e., the optimal printing distance between the nozzles and the substrate. Getting far from optimal stand-off, where the laser beam and the powder flow are focused, impairs the printed part quality, especially from a geometrical point of view. In the present work, an innovative process monitoring approach has been investigated. Sensors such as a pyrometer and RGB camera have been integrated into a DED machine, to measure the radiation emitted by the melted material at different spectral bandwidths. Since variations in stand-off distance affect the signals acquired, a neural network approach has been derived to estimate the stand-off of the deposition at each layer and highlight possible anomalous situations. The training dataset has been acquired by printing tracks at different stand-offs. The proposed solution has been validated by comparing the stand-off estimated on defective multilayer parts with the measurements provided by a high-resolution 3D scanner. Preliminary results show that the proposed method is a promising approach to prevent critical stand-off deviations.

Deep reinforcement learning (DRL) is based on rigorous mathematical foundations and adjusts network parameters through interactions with the environment. The stability problem of maintaining a vehicle on a continuous path can be achieved by soft actor-critic (SAC). Furthermore, a model predictive control (MPC) with prediction and control horizons under multivariable constraints can precisely follow the path, but the disadvantage is its large computation. In this paper, a DRL control scheme with MPC is proposed to precisely and effectively implement the path following and obstacle avoidance of tracked vehicle. The DRL controller performs the effective obstacle avoidance and is also in accordance with MPC to precisely follow planning paths. To make the training more realistic, a data-driven state-space dynamic model of the tracked vehicle is first estimated via N4SID system identification algorithm. During the DRL training, the MPC output is used as the reward input of the DRL to learn the MPC characteristics and an additional reward function is designed specifically for obstacle avoidance. The parameters of the DRL agent are adjusted based on the environment input and the MPC output. After the training, the MPC can be skipped since it is used as a part of the reward function, and the DRL has learned to imitate the MPC while achieves obstacle avoidance. The simulation and experimental results show that the overall controller has high stability, accuracy, and efficiency.

Multispectral LiDAR is an emerging active remote sensing technique that combines distance and spectroscopy measurements on light reflected from the surface at the respective measurement point. It is known that the reflectance spectrum can be used for material classification. However, the spectrum also depends on other surface parameters, particularly surface roughness. Herein, we propose an extension of multispectral to polarimetric multispectral LiDAR and introduce polarized and unpolarized reflectance spectra as additional features for classifying materials and roughness. We demonstrate the feasibility and the benefit using a bench-top prototype instrument which allows acquiring standard, polarized and unpolarized reflectance spectra, in addition to distance, in 33 spectral channels with 10 nm bandwidth between 580 and 900 nm. We analyze and interpret the raw spectra obtained from measurements on test specimens consisting of five different materials (PE, PVC, PP, sandstone, limestone) with two different levels of surface roughness. Using a linear support vector machine (SVM) we demonstrate the potential of the different features for independent material and roughness classification. The results indicate that the unpolarized reflectance spectrum increases the material classification accuracy by 50% as compared to a standard spectrum, and that the polarized spectrum actually allows classifying roughness. We interpret the results as a strong indication that multispectral polarimetric LiDAR enables deriving practically relevant additional information on surfaces with high spatial resolution through remote sensing.

Carbon Fiber Reinforced Plastic (CFRP) is an important material in manufacturing, in particular in automotive and aerospace industry. Its relevance is due to its lightness, resistance and rigidity. In particular the lightness/resistance ratio is the property that will give to CFRP a favourable future in more and more applications, especially where the lightness and resistance are essential. On the other end CFRP is expensive and difficult to produce. During the production process of CFRP’s artefacts some defects may occur (e.g. material inclusions or bubbles), making the component not usable. When applicable, CFRP repair process is structured in: defect removal (scarfing), cleaning and patch application. Actually, scarfing is a slow process manually performed by human operators, strongly dependent on the workers’ skills, that generates a lot of toxic nano dusts. The present paper deals with the implementation of a (semi)automatic robotic cell to repair large CFRP aeronautic components. The robotic cell is composed by a collaborative robot equipped by a custom scarfing tool due to the context-specific constraints to be implemented. Under these constraints, the machining got is not as perfect as in a traditional milling process, thus requiring additional quality control and inspection after scarfing takes place. The paper describes the designed scarfing tool, examines the trajectories generated to execute the scarfing and illustrates some preliminary results in terms of accuracy of the shape got by the designed tool.

This paper presents a preliminary study for evaluating the quality of welds in thermomagnetic switches using 3D sensing and machine learning techniques. A 3D sensor based on laser triangulation is used to gather the point cloud of the component. The point cloud is then processed to extract hand-crafted signatures for binary classification: defective or non-defective component. Features such as Gaussian and mean curvatures, density, and quadric surface properties, are used for building these significant signatures. Different machine learning models, including decision trees, Support Vector Machines, k-nearest neighbors, random forests, ensemble classifiers, and Artificial Neural Networks, are trained using the built signatures to classify the weld as defective or non-defective. Preliminary results on actual data achieve high classification accuracy (<84%) on all the tested models.

Monitoring and maintaining the health state of existing bridges is a time-consuming and critical task. To reduce the time and effort required for a first screening to prioritize risks, deep-learning-based object detectors can be used. In detail, automatic defect and damage recognition on existing elements of existing bridges can be performed using single-stage detectors, such as YOLOv5. To this end, a database of typical defects was gathered and labeled by domain experts and YOLOv5 was trained, tested, and validated. Results showed good effectiveness and accuracy of the proposed methodology, opening new scenarios and the potentialities of artificial intelligence for automatic defect detection on bridges.

We are developing a lightweight Raman imager for detecting harzardous chemical compounds on surfaces in emergency settings such as industrial accidents, terrorist attacks, and chemical warfare. Our instrument is based on a compact Fourier-Transform hyperspectral sensor which records interferograms used as signatures of chemicals compounds. Two important challenges relating to the detection of small amounts of chemicals with our device are (i) crosstalk artefacts degrading the reproducibility of the measurements, and (ii) the mixture of signatures from target compound and substrate material. We report on advances on crosstalk correction, and chemical detection by linear unmixing of the signatures. The system is demonstrated on the detection of two organophosphorus compounds on PMMA, a polymer with mild Raman signal.

Mapping of small and shallow water bodies is still done primarily in a manual, labour-intensive process. The Fraunhofer Institute for Physical Measurement Techniques IPM developed and evaluated a new compact and lightweight airborne laser-scanning system targeted at topographic and bathymetric surveys which can greatly increase the efficiency and precision. The sensor weighs less than 4kg and can therefore be operated on many commercially available drones without special permissions according to current regulations. This is a decisive aspect in achieving the necessary operational capability and economic efficiency of such a system. To optimally address the intended use case, our sensor simultaneously emits green and infrared light. This allows for precise separation of water surface and seabed and is especially benefi cial for increasing precision in shallow water regions and accurately detecting the shoreline. For sophisticated data analysis, the system records and saves the full signal waveform using high-speed analog-to-digital converters. To ensure a safe operation and reduce the required safety measures, the system is designed as a class 2M laser product. This is achieved through expanding and collimating the outgoing laser beam to approximately 50mm in diameter and therefore reducing the light intensity. A downward-facing palmer scanner deflects the beam in a near-elliptical shape at approximately 15° off nadir. The system has been tested and evaluated at a quarry lake in turbid water conditions. The result is a high-resolution ground model which demonstrates the bene fits and feasibility of a compact bathymetric scanner and highlights the potential for further research in this field.

Caterpillars have been causing havoc to agriculture due to feeding aggressively on the foliage of the crops. The current methods of pest control like sticky traps or pheromone traps work on adult moths, but not on foraging caterpillars. Chemical means like pesticides are effective, but chemical residues on crops are concerning. Therefore, this study aims to primarily track and estimate the 3D position of the caterpillars in orchards in real-time. To this end, we have employed the state-of-the-art object detector YOLOv7, combined with Simple Online and Realtime Tracking with a Deep Association Metric (DeepSORT) algorithm. This combined approach, when compared to merely YOLOv7, has improved detection up to 25%, courtesy of the SORT embedded tracker. The RGB-D camera is utilized for this work is Intel Realsense D405. For the training data, 2,000 images captured in a jujube orchard with varying exposure, occlusion, and wind conditions were used. Inference was done from completely new images in real time. In the experiments, the YOLOv7+SORT approach makes detections within 17 ms per frame, with an average detection rate of 85%, indicative of is real-time applicability in orchards. The smallest object (around 2-cm length caterpillar) is recognized around 21×12 pixels, which is at a distance of 35 cm from the camera. Thus, this development of YOLOv7+SORT approach can be integrated with technologies like robot arms, that can pick the caterpillars, or even into stand-off techniques like laser pest targeting, which can help eradicate the pest problems in a physical manner efficiently.

This study addresses a 2D scalar electromagnetic inverse source problem by using a deep neural network-based artificial intelligence technique. Specifically, the Learned Singular Value Decomposition (L-SVD) approach based on hybrid autoencoding is adopted. The main goal is to reproduce the singular value decomposition through neural networks and compare the reconstruction performance of L-SVD and truncated SVD (TSVD) in the case of noiseless data, which represents a reference benchmark. The results demonstrate that L-SVD outperforms TSVD in terms of spatial resolution.

Multimodal sensing is presented based on the measurement of density oscillations in swirl-stabilized flames, where density is coupled to physical quantities like heat release rate or sound pressure and advection velocity. The measurement approach is a high-speed camera based interferometer combined with a multi-camera background oriented Schlieren system, both enabling line of sight detection and therefore require solution of the inverse problem by tomographic reconstruction. Those reconstructions are realized using a neural network, resulting in the three-dimensional distribution of local thermoacoustic oscillations. Finally, the advection velocity of those oscillating vortex structures is calculated by image correlation.

We demonstrate a supercontinuum-based hyperspectral laser scanning technique that provides high-precision distance measurements of natural surfaces along with their reflectance signature over the broad spectral range of the supercontinuum (SC) output. The SC used in our experiments is spectrally broadened to 570-970 nm from a 780 nm mode-locked femtosecond laser. Distance measurements are carried out by monitoring the differential phase delay of the intermode beat notes obtained from direct photodetection of the SC, while the backscattered reflection spectrum is acquired using a commercial spectrometer. We achieve a single-point range precision below 10 μm on natural targets (gypsum board and leaves of a plant used herein) placed at a stand-off distance of 5 m. Our results demonstrate the acquisition of hyperspectral point clouds together with sub-mm range noise on the scanned surface. This range performance is comparable to commercial state-of-the-art terrestrial laser scanners which traditionally employ a monochromatic laser source.We show the benefit of enhanced range precision toward correctly estimating the surface orientation and for radiometric calibration of the acquired intensities. Initial results illustrate the direct 3D mapping of spectral data of plant leaves with a reduced angle of incidence-related bias, highlighting new opportunities for future research into remote sensing of vegetation.

In recent times, there has been a surge of interest in LiDAR imaging systems, particularly in outdoor terrestrial applications associated with computer vision. However, a significant hurdle preventing their widespread implementation lies in their limited tolerance for adverse weather conditions, such as fog. To address this challenge, researchers have explored the capability of polarization in improving detection capabilities in such media. This paper explores the potential of LiDAR technology to obtain polarized images through fog and investigates the impact of fog on object detection using digitized temporal signals and point clouds. The study utilizes a LiDAR-polarized imaging system using circular polarization, which has been shown to enhance image contrast in highly-dispersive media. The analysis of the polarimetric information of the backscattered light signal in fog reveals its influence on object detection and evaluates the range difference between orthogonal polarimetric channels: coplanar and cross-configuration. The results demonstrate that cross-configuration detection provides larger range and more detailed point clouds compared to co-planar configuration, particularly benefiting metallic objects, for the same foggy conditions. By utilizing circularly polarized incident light and cross-configuration detection, the LiDAR system can improve the signal-to-noise ratio by filtering out the co-polarized fog responses. However, the range of the system may be reduced compared to non-polarized detection. Overall, our findings indicate that utilizing a cross-polarization detection setup can effectively reduce the impact of fog backscatter while preserving the return signal from objects of interest in the majority of cases.

Non-contact methods can expand the application scenarios of blood oxygen measurement with better hygiene and comfort, but the traditional non-contact methods are usually less accurate. In this study a novel non-contact approach for measuring peripheral oxygen saturation (SpO₂) using deep learning and near-infrared multispectral videos is proposed. After a series of data processing including shading correction, global detrending and spectral channel normalization to reduce the influences from illumination non-uniformity, ambient light, and skin tone, the preprocessed video data are split into half-second clips (30 frames) as input of the 3D convolutional residual network. In the experiment, multispectral videos in 25 channels of hand palms from 7 participants were captured. The experimental results show that the proposed approach can accurately estimate SpO₂ from near-infrared multispectral videos, which demonstrates the agreement with commercial pulse oximeter. The study also evaluated the performance of the approach with different combinations of near-infrared channels.

The nature of phase retrieval algorithms in holographic tomography (HT) is the reason that, in order to achieve the refractive index (RI) reconstruction, phase unwrapping (PU) has to be carried out beforehand. Conventional PU algorithms present shortcomings regarding the speed, noise robustness, reliability, and automation possibilities. These problems can be tackled by implementation of deep learning (DL) and convolutional neural networks in the PU process. The work presents a two-step pipeline consisting of denoising and unwrapping models. Both steps are realised under the umbrella of a U-Net architecture; however, unwrapping, which is carried out via semantic segmentation, is aided by Attention Gates and Residual Blocks (RB) introduced to the model. Through the experiments, the proposed pipeline is capable of denoising and unwrapping highly irregular and complex phase images captured in HT. What is more, the pipeline is implemented into a tomographic reconstruction algorithm. The quality of the final RI distribution is evaluated against the reconstruction with the conventional State-of-the-Art solution. To our knowledge, this is the first work, that puts forward phase unwrapping via segmentation with an Attention U-Net with RBs, assisted by a pre-processing denoising step. Moreover, this is the first solution, based on DL, that is trained entirely on experimental phase images captured in HT, and whose performance is evaluated on the distribution of RI of the measured specimen.

Cellular populations are often heterogeneous, thus the intraspecies variability is usually lost in the average measurements accessed by conventional measuring devices. For this reason, the single-cell analysis provided by optical microscopy is opening new promising perspectives in biomedicine. The gold standard technique in this context is Fluorescence Imaging Flow Cytometry (FIFC), as big datasets of single cells flowing in suspension through a measuring device can be collected in short times and multiple parameters can be measured at the single-cell level. However, to overcome the limitations related to the staining process, in the recent years Holographic Imaging Flow Cytometry (HIFC) has been proved to be a viable label-free alternative to FIFC, able to give access also to the cell biophysical features. Very recently, the latest evolution of HIFC has been demonstrated, i.e. Tomographic Phase Imaging Flow Cytometry (TPIFC), in which the 3D spatial distribution of the refractive indices of single cells can be reconstructed thanks to their roto-translation along a microfluidic channel. However, to retrieve one single tomogram, hundreds of digital holograms must be converted into the corresponding phase-contrast maps. Currently, this is the actual bottleneck for the high-throughput TPIFC, which aims to a fast 3D analysis of the cellular populations. Therefore, here we show that a fully convolutional end-to-end context aggregation neural network can greatly speed up the phase retrieval process, thus reducing the computational time for the tomographic reconstruction from tens of minutes to few seconds, while providing at the same time high fidelity and small memory footprint.

The term Metaverse was introduced in 1992. Lately, the concept has become a popular buzzword in the general public, thanks to Meta. And yet, its intrinsic reliance on the convergence between enabling technologies such as virtual reality, the internet, and social networks cannot be understated. It might prove a powerful tool for enhancing data discovery and interpretation, especially considering a collaborative setup. The work presented here aims to investigate the interaction between a real user and digital objects in a virtual world to understand which aspects of attention must be focused on to obtain a natural and comfortable use. Our work involved the generation of Metaballs and the interaction with them by a user in a virtual environment. Metaballs are particular implicit surfaces of arbitrary typology, widely used in Computer Graphics to model curved objects. The ”organic” look and feel of how they interact with each other and their resemblance to soft tissues have proved a natural fit for tasks such as surgery simulation. Still, these implicit surfaces can represent even real objects at an even smaller scale, like cells of living organisms, and the ability to comfortably interact with this supersized version of real objects could open the door to new possibilities.

The analysis of innovative materials and processes stands at the frontier of a series of wide-ranging scientific problems and poses stimulating challenges from a scientific as well as technological point of view, by virtue of its connection with various industrial sectors, such as aerospace and aeronautics. In recent years, composite materials have found numerous applications due to their mechanical characteristics and properties, representing the evolution of materials science and technologies by fusing within them the best characteristics of multiple materials. The present work is focused on the characterization of composite materials, using non-destructive techniques (NDT), to check different kinds of defects eventually present for a quality control of the object under observation. Shearography and thermography are used as nondestructive methods. The former, is an optical interferometric method for the detection of surface or sub-surface defects, the latter is a diagnostic technique that, by measuring the infrared radiation emitted by a body, allows to determine its surface temperature and to understand the health status of the investigated object. The results of the shearography technique, including, are complementary to thermographic techniques and allow us to have a complete characterization of the object. Their use offers advantages related to visualization and testing of end products, as well as the noncontact nature, nondestructive and areal working principle, rapid response, high sensitivity, resolution, and accuracy.

Lensless microscopy is an emerging imaging modality that overcomes the inherent limitation of conventional lens-based optics, especially in terms of imaging throughput, functionality, and cost-effectiveness. Pixel super-resolution phase retrieval serves as the key underlying technique for reconstructing high-resolution holographic images from the raw measurements. In this talk, we revisit lensless microscopy from a computational imaging perspective. A unified mathematical framework is established and the encoding and decoding mechanisms of the phase and subpixel information are analyzed. Regularization and Nesterov’s momentum techniques are introduced to speed up the data acquisition and reconstruction procedures, respectively. The proposed algorithms are verified through a proof-of-concept lensless on-chip microscope. We experimentally demonstrate the capability of pixel super-resolution phase retrieval techniques in revealing the subpixel and quantitative phase information of complex biological samples.

Microplastics (MPs) are not degradable pollutant, yet curbing their consumption to safeguard marine waters is not an easy task. Plastic fragmentation makes detection and discrimination of micron sized elements a difficult task, decelerating plastic recovery policies, especially in marine waters. MPs of all types and shapes are subject to continuous research for defining standard identification protocols. Among the most recent methodologies, digital holography (DH) demonstrated to be an efficient and reliable imaging tool for MPs discrimination. DH furnishes quantitative information of samples in a label-free mode and with flexible refocusing capabilities. DH refocuses the probed object in post-processing by using numerical solutions of the diffraction integral, and can be designed to be compacted for high-throughput field portable microfluidic systems. Here we show the fruitful combination of DH and artificial intelligence to identify and characterize micro-plastics. We show how fractal geometry applied to DH phase-contrast maps plays an important role in providing a set of features that well describe MPs and micro-plankton. Classification results prove the reliability of DH fractal features. Besides, we show that adding multiple information channels to a DH system allows mimicking material specificity at a certain extent. In particular, we discriminate between different types of synthetic and natural microfibers by combining AI and a polarization-resolved DH system. We distinguish microfibers by training several classifiers using features calculated from the Jones matrix and birefringence features. Results corroborate the use of conventional machine learning applied to holographic maps to solve a variety of classification issues in water pollution assays.

EUV ptychography combines high resolution, strong material contrast, reasonable penetration depth, and easy sample preparation. It thus has the potential to bridge the gap between visible light and electron microscopy. In this contribution, we present recent results on table-top ptychographic EUV microscopy. The experimental setup relies on a high photon flux 13.5 nm wavelength high-order harmonic source. A simple amplitude mask upstream of the sample tailors the illumination. The sample is scanned by piezo-driven stages. Finally, far-field diffraction patterns are recorded by an EUV detector and fed into a ptychographic iterative engine to retrieve both the illumination and the sample transmission in amplitude and phase. The resolution of the EUV microscope (16 nm) was characterized using a Siemens star test sample. In the next step, a thin lamella of an integrated circuit was investigated. The resulting EUV images exhibit a rich diversity in amplitude and phase. We analyzed that the relative amplitude precision is better than 4% and the absolute phase precision as good as ~ 20 mrad. In this way, the microscope provides excellent input data for further analysis of the material composition. For this purpose, the projected scattering quotient is calculated from the reconstructed phase and amplitude and compared to tabulated material parameters. Here, materials like Al, Si₃N₄, and SiO₂ were determined with high sensitivity. Finally, biological samples, namely germlings of the filamentous fungus Aspergillus nidulans were investigated. A similar scattering quotient analysis allowed us to clearly distinguish regions with low- and high lipid and phospholipid concentrations.

Achieving clear vision through smoke and flames is a highly pursued goal to better manage intervention priorities and to allow first responders operating safely during fire accidents. Here we show active far-infrared systems to image static/moving targets through fire with different imaging performance and field-portability characteristics. Low-coherence infrared systems and high-coherence holographic sensors will be discussed. We show that a pre-trained convolutional neural network can detect the presence of a person hidden behind fire in real-time, accurately, even when the system is not able to reject the flame contributions in full, being suitable for video-surveillance applications.

The assessment of microplastics (MPs) pollution and water quality monitoring raise a lot of attention in recent years. Discriminative methods are highly needed for quick and accurate in situ MP detections. Digital holography records the wavefront information of the objects and contains the morphology, refractive index, and roughness information. Polarization imaging inspects the optical anisotropy of MPs, which is related to their birefringence and material characteristics. In this work, we explore the capability of holographic and polarization imaging for the identification of MPs. The computed features, such as the angle of polarization (AoP) and degree of linear polarization (DoLP), show distinguishable characteristics of MPs. We inspect the method feasibility on MP classification as well as biological and natural particles. The proposed method shows potential use in real-time, non-contact in situ MPs detection and water pollution monitoring.

Weld quality inspection allows the detection of defects that may compromise the quality and strength of the weld. Although visual optical inspection offers lower reliability than other non-destructive methods, it enables weld analysis at a significantly lower cost. In this context, developing machine learning-based algorithms for automatic optical weld quality recognition requires acquiring large amounts of data for training. This entails high costs in terms of time, material and energy required for test preparation. However, one possible approach to tackling the problem with limited datasets is to use synthetic data. Using such data increases the amount and variety of data available to the detection algorithm. With a focus on the context of welding, this paper presents an approach that uses synthetic data as a form of data augmentation to improve the performance of the optical detection of weld seams. Specifically, we propose a generative neural network for semantic image synthesis using a limited starting dataset. The network generates new data instances by receiving as input a semantic map of the image to be represented. Weld defects such as porosity or weld spatter are added to the semantic map so that the network synthesizes corresponding defect images. Analysing the performance on a segmentation network, experimental results show how adding synthetic data to the original data can ensure improvements in network performance.

In electronics manufacturing, the inspection of defects of electrical components on printed circuit boards (SMD-PCB) is an import part of the production chain. This process is normally implemented by automatic optical inspection (AOI) systems based on classical computer vision and multimodal imaging. Despite the highly developed image processing, misclassifications can occur due to the different, variable appearance of objects and defects and constantly emerging defect types, which can only be avoided by constant manual supervision and adaption. Therefore, a lot of manpower is needed to do this or to perform a subjective follow-up. In this paper, we present a new method using the principle of multimodal deep learning-based one-class novelty-detection to support AOIs and operators to detect defects more accurate or to determine whether something needs to be changed. By combining with a given AOI classification a powerful adaptive AOI system can be realized. To evaluate the performance of the multimodal novelty-detector, we conducted experiments with SMD-PCB-components imaged in texture and geometric modalities. Based on the idea of one-class-detection only normal data is needed to form training sets. Annotated defect data which is normally only insufficiently available, is only used in the tests. We report about some experiments in accordance with the consistence of data categories to investigate the applicability of this approach in different scenarios. Hereby we compared different state-of-the-art one-class novelty detection techniques using image data of different modalities. Besides the influence of different data fusion methods are discussed to find a good way to use this data and to show the benefits using multimodal data. Our experiments show an outstanding performance of defect detection using multimodal data based on our approach. Our best value of the widely known AUROC reaches more than 0.99 with real test data.

In-field sensing systems for automatic yield monitoring are gaining increasing importance as they promise to give a considerable boost in production. The development of artificial intelligence and sensing technologies to assist the human workforce also meets sustainability needs, which impact the ecological goals of current and future agricultural processes. In this context, image acquisition and processing systems are widely adopted to extract useful information for farmers. Although RGB-D cameras have been used in many applications for ground-based proximal sensing, relatively few works can be found that include depth information in image analysis. In this work, both semantic and depth information from RGB-D vineyard images is used in processing pipeline composed of a decision tree algorithm and a deep learning model. The goal is to reach coherent semantic segmentation of a set of natural images acquired at both long and short distances, using a low-cost RGB-D camera in an experimental vineyard. Depth information of each image is fed into a decision tree to predict the distance of the acquired vines from the camera. Before feeding the deep learning models, the images to be segmented are manipulated according to the predicted distance. The results of semantic segmentation with and without using the decision tree are compared, showing how depth information appears to be highly relevant in enhancing the accuracy and precision of the predicted semantic maps.

Stress-strain detection of scattering materials, such as metal surfaces, is an unavoidable issue during bolt assembly monitoring; conventional strategies of laser morphology will be limited by scattering illumination. Contact and marked measurement have been considered as acceptable solutions for long time. Herein, we implement the multimodal characterization for bolted joint surface. A reflection digital holographic microscope (RDHM) has been combined with laser triangulation measurement for achieving quantitative micro-deformation description of the connecting region between bolts and metal plate. The calibrated laser triangulation instrument is installed on the electronically controlled translation platform for two-dimensional scanning. A structure extraction algorithm is proposed to realize fast 3D reconstruction of the scanned area. The profiles obtained by triangulation are used, as ground truth, to calibrate the measured reflective holographic morphology. Affected by the strong scattering from the metal surface, it is difficult to completely characterize the metal surface by the single shot holographic measurement results. But considering that the spatial resolution requirement of strain measurement is low during bolt installation, holographic measurement will be an excellent compromise with high temporal resolution. In this case, the triangulation will complement digital holography well. Therefore, the strategy of mixing the two methods will become a preferred solution in near future.

This paper describes a novel image acquisition and processing framework to detect and count grape bunches along vineyard rows, using RGB and depth data acquired in the field by a farmer robot. The proposed pipeline starts with a semantic image segmentation module that uses a pre-trained convolutional neural network to separate fruit from non-fruit regions. Areas pertaining to fruits are then further processed using a depth gradient-based clustering algorithm to detect and separate single grape bunches. Experiments performed in a commercial vineyard are presented showing that, despite the low quality of the input images, the proposed approach is able to correctly detect and count grape clusters with good accuracy.

Conventional (CP) and Fourier (FP) ptychography have emerged as versatile quantitative phase imaging techniques. While the main application cases for each technique are different, namely lens-less short wavelength imaging for CP and lens-based visible light imaging for FP, both methods share a common algorithmic ground. CP and FP have in part independently evolved to include experimentally robust forward models and inversion techniques. This separation has resulted in a plethora of algorithmic extensions, some of which have not crossed the boundary from one modality to the other. Here, we describe the principle of reciprocity in ptychography. This principle allows for flexible conversion between CP and FP. Moreover, we present an open source, crossplatform software, called PtyLab, enabling a unified CP and FP data analysis. With this framework, we aim to facilitate and accelerate cross-pollination between the two techniques. Moreover, the availability in Matlab, Python, and Julia will set a low barrier to enter each field.

Fiducial markers are used in a broad spectrum of applications to tag objects visually and estimate their pose using a camera sensor. Their flexibility, simplicity, and cost-effectiveness make them relevant in research setups and experiments. While the detection and pose estimation frequency are related to the computation capability of the used machine; high frame-rate cameras and high-resolution cameras are shown to harden the recognition rate. To this aim, we provide a dynamic region-of-interest(ROI) based filter for the inputted images in order to decrease the computational load to cope with high throughput systems. To do so, we leverage the kinematic properties of the objects tagged with the fiducial marker and use an Unscented Kalman Filter (UKF) to update its state.

To eliminate the noise component in infrared images, the paper proposes a method of inter-frame data alignment, built on the operations of pre-processing and highlighting the contours of objects and based on the following basic stapes: at the initial stage, the system operation parameters are determined, the frame size is read, the color field depth and speed fixing frames; the operation of selecting local areas with minimization of objects allows you to create a mask of objects and localize subsequent methods; determining the construction of inter-frame pixel shift vectors; accumulationformation of color intensity change data; data filtering based on a multi-criteria method based on the application of the criterion of the standard deviation of the results and the generated estimates, as well as the standard deviations of the final differences of the second order; frame processing of images by a two-dimensional multi-criteria method with preservation of object boundaries; framing the output image. On the set of test data obtained by the seak and flir cameras, examples of processing objects of simple shapes are given. Examples of visual reduction of the noise component on frames of test images are shown. Examples of error reduction in the formation of the proposed approach on synthetic sequences with superimposed Gaussian noise are given.

For both economic and research purposes, accurate information on forest composition, and the amount of dead wood is of paramount importance. It is not only important to know the biomass and the distribution of tree species but also to detect vermin and diseases to assess the health of a forest. Performing such inventory accurately with conventional methods of surveying (e.g., terrestrial laser scanners) is very labor and time-consuming as forests can be very dense, thus requiring many setups with a terrestrial laser scanner. We present an innovative approach for forest inventory combining UAV-borne light detection and ranging (LiDAR), multispectral aerial imagery, and terrestrial point cloud data measured with a handheld laser scanner. For such multi-sensor measurement campaigns, however, reliable extraction of forest parameters such as canopy height, the diameter at breast height (DBH), deadwood volumes, etc., strongly relies on the quality of sensor data fusion. Especially in dense forests, the GNSS signal, which is necessary for georeferencing the point clouds, can be very weak. However, terrestrial laser scans can capture much more information underneath the forest canopy which is partly obscured to the airborne data. To circumvent this mismatch, we propose an easily adaptable two-step workflow for fusing the directly georeferenced airborne LiDAR point clouds with the corresponding multi-spectral photogrammetric data and their unreferenced terrestrial counterparts. In the first step of the processing chain, individual terrestrial scans of the forest are coregistered using spherical laser targets located above exactly measured reference points. Secondly, these coarsely coregistered scans are then combined with the georeferenced airborne point clouds using control points. This is especially challenging as forests are very unstructured environments, and in addition, typical SLAM-features such as leaves, and branches tend to move. The result of this procedure are radiometrically enhanced 3D point clouds containing detailed information about the forest structure above and underneath its canopy. This provides a versatile basis for further processing such as tree species recognition, e.g., using convolutional neural networks. We demonstrate the applicability of the workflow on recent data sets measured in Landshut (Bavaria, Germany) under leaf-on and leaf-off condition.

This paper investigates the potential of a consumer-grade infrared stereo camera, i.e. the Intel RealSense D435, to automatically extract crop status information, such as Normalized Difference Vegetation Index (NDVI), in arable and permanent crops. The sensing device includes two infrared (IR) sensors for depth calculation and one colour sensor, which provide, for each point of the scene, both IR and visible light information thus making it possible pixel per pixel NDVI estimations. Measurements were performed on various arable crops including corn (Zea mays) and barley (Ordeum vulgare) and on two vine varieties, Freisa and Malvasia, and were compared to measurements taken by a Trimble GreenSeeker handheld crop sensor. Results show that the RealSense camera tends to underestimate NDVI values compared to the GreenSeeker, with squared correlation coefficient r2 = 0.68. The fitted regression equation is successively applied to correct new camera observations, resulting in good agreement with the GreenSeeker output. The use of the RGB-D camera to simultaneously provide canopy height measurements by a farmer robot is also demonstrated in a Malvasia field, showing that the proposed system can be effectively adopted for fully automated plant-scale monitoring of vineyards.

A microcontroller-based multi-sensor system using ultrasonic range sensors and radar sensors for sensor data fusion is presented. Using this system as a warning system can detect and prevent forklift accidents in the industry: The system uses the combined sensor data to detect objects’ distance, angle, and speed and calculate object trajectories to determine the risk of each moving object in the detection area. Because the dimensions of the microcontroller-based warning system are very small, it can be integrated into a warning vest for a factory worker. If the system detects and categorizes objects as risk, the factory worker is noticed by vibrotactile feedback about the location of the risk object. The paper deals with the introduction of the system followed by accuracy study and performance test.

Automation of production processes using robots is a priority for developing many industrial enterprises. Human-machine interaction is a key component of such control infrastructure. The proposed algorithm is a four-stage procedure: (a) fusion information from multimodal sensors based on the quaternion model, (b) image preprocessing using a 3D Gabor filter, (c) a descriptor calculation using 3D local binary dense micro-block difference with skeleton points, and (d) classification. The proposed algorithm is based on capturing 3D sub-volumes located inside a video sequence patch and calculating the difference in intensities between these sub-volumes; for intensified motion, used the convolution with a bank of 3D arbitrarily oriented Gabor filters and calculating 3D local binary dense micro-block difference. A program was developed for transmitting information about the target points of the robot's movement through a virtual TCP / IP port and a script for working out the target points in the simulation environment. To test the effectiveness of the proposed algorithm, we simulate an action recognition system for Human-robot collaboration in the RoboGuide environment.