Dokumentdetails
ID
oai:arXiv.org:2403.07432
Thema
Computer Science - Computer Vision...Autor
Zhou, Hanyu Chang, Yi Shi, Zhiwei Yan, LuxinKategorie
Computer Science
Jahr
2024
Auflistungsdatum
20.03.2024
Schlüsselwörter
complementary knowledge features motion space visual sceneZusammenfassung
Single RGB or LiDAR is the mainstream sensor for the challenging scene flow, which relies heavily on visual features to match motion features.
Compared with single modality, existing methods adopt a fusion strategy to directly fuse the cross-modal complementary knowledge in motion space.
However, these direct fusion methods may suffer the modality gap due to the visual intrinsic heterogeneous nature between RGB and LiDAR, thus deteriorating motion features.
We discover that event has the homogeneous nature with RGB and LiDAR in both visual and motion spaces.
In this work, we bring the event as a bridge between RGB and LiDAR, and propose a novel hierarchical visual-motion fusion framework for scene flow, which explores a homogeneous space to fuse the cross-modal complementary knowledge for physical interpretation.
In visual fusion, we discover that event has a complementarity (relative v.s. absolute) in luminance space with RGB for high dynamic imaging, and has a complementarity (local boundary v.s. global shape) in scene structure space with LiDAR for structure integrity.
In motion fusion, we figure out that RGB, event and LiDAR are complementary (spatial-dense, temporal-dense v.s. spatiotemporal-sparse) to each other in correlation space, which motivates us to fuse their motion correlations for motion continuity.
The proposed hierarchical fusion can explicitly fuse the multimodal knowledge to progressively improve scene flow from visual space to motion space.
Extensive experiments have been performed to verify the superiority of the proposed method.
Zhou, Hanyu,Chang, Yi,Shi, Zhiwei,Yan, Luxin, 2024, Bring Event into RGB and LiDAR: Hierarchical Visual-Motion Fusion for Scene Flow
Configurable Safety Tuning of Language Models with Synthetic Preference Data
data preference cst dpo configurable language safety
Affective flexibility as a developmental building block of cognitive reappraisal: An fMRI study
cognitive success affective flexibility
