Attention-based cross-modality interaction for multispectral pedestrian detection

Tianshan Liu; Rui Zhao; Kin-Man Lam

doi:10.1117/12.2590661

13 March 2021 Attention-based cross-modality interaction for multispectral pedestrian detection

Tianshan Liu, Rui Zhao, Kin-Man Lam

Proceedings Volume 11766, International Workshop on Advanced Imaging Technology (IWAIT) 2021; 1176603 (2021) https://doi.org/10.1117/12.2590661
Event: International Workshop on Advanced Imaging Technology 2021 (IWAIT 2021), 2021, Online Only

Abstract

Multispectral pedestrian detection has attracted extensive attention, as paired RGB-thermal images can provide complementary patterns to deal with illumination changes in realistic scenarios. However, most of the existing deep-learning-based multispectral detectors extract features from RGB and thermal inputs separately, and fuse them by a simple concatenation operation. This fusion strategy is suboptimal, as undifferentiated concatenation for each region and feature channel may hamper the optimal selection of complementary features from different modalities. To address this limitation, in this paper, we propose an attention-based cross-modality interaction (ACI) module, which aims to adaptively highlight and aggregate the discriminative regions and channels of the feature maps from RGB and thermal images. The proposed ACI module is deployed into multiple layers of a two-branch-based deep architecture, to capture the cross-modal interactions from diverse semantic levels, for illumination-invariant pedestrian detection. Experimental results on the public KAIST multispectral pedestrian benchmark show that the proposed method achieves state-of-the-art detection performance.

Citation Download Citation

Tianshan Liu, Rui Zhao, and Kin-Man Lam "Attention-based cross-modality interaction for multispectral pedestrian detection", Proc. SPIE 11766, International Workshop on Advanced Imaging Technology (IWAIT) 2021, 1176603 (13 March 2021); https://doi.org/10.1117/12.2590661

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available