Action recognition algorithm from multimodal images for human-robot collaboration systems

V. Voronin; M. Zhdanova; E. Semenishchev; A. Zelensky; N. Gapon; Yu. Ilyukhin

doi:10.1117/12.2678112

9 August 2023 Action recognition algorithm from multimodal images for human-robot collaboration systems

V. Voronin, M. Zhdanova, E. Semenishchev, A. Zelensky, N. Gapon, Yu. Ilyukhin

Proceedings Volume 12621, Multimodal Sensing and Artificial Intelligence: Technologies and Applications III; 126211A (2023) https://doi.org/10.1117/12.2678112
Event: SPIE Optical Metrology, 2023, Munich, Germany

Conference Poster

Abstract

Automation of production processes using robots is a priority for developing many industrial enterprises. Human-machine interaction is a key component of such control infrastructure. The proposed algorithm is a four-stage procedure: (a) fusion information from multimodal sensors based on the quaternion model, (b) image preprocessing using a 3D Gabor filter, (c) a descriptor calculation using 3D local binary dense micro-block difference with skeleton points, and (d) classification. The proposed algorithm is based on capturing 3D sub-volumes located inside a video sequence patch and calculating the difference in intensities between these sub-volumes; for intensified motion, used the convolution with a bank of 3D arbitrarily oriented Gabor filters and calculating 3D local binary dense micro-block difference. A program was developed for transmitting information about the target points of the robot's movement through a virtual TCP / IP port and a script for working out the target points in the simulation environment. To test the effectiveness of the proposed algorithm, we simulate an action recognition system for Human-robot collaboration in the RoboGuide environment.

Citation Download Citation

V. Voronin, M. Zhdanova, E. Semenishchev, A. Zelensky, N. Gapon, and Yu. Ilyukhin "Action recognition algorithm from multimodal images for human-robot collaboration systems", Proc. SPIE 12621, Multimodal Sensing and Artificial Intelligence: Technologies and Applications III, 126211A (9 August 2023); https://doi.org/10.1117/12.2678112

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available