arXiv · 2025 · Preprint
SPKLIP: Aligning Spike Video Streams with Natural Language
SPKLIP aligns asynchronous spike video streams with natural language through hierarchical multiscale temporal modeling and spike-text contrastive learning.
The scientific question
How can sparse, high-speed spike video streams be aligned with natural language?
Standard vision-language models like CLIP struggle on spike camera outputs because frame-based encoders mismatch sparse, microsecond-resolved event streams.
The core idea
Hierarchical Spike-Text Contrastive Alignment
SPKLIP extracts multiscale temporal dynamics from spike streams via a hierarchical encoder (including a full-spiking variant) and aligns visual features with text through contrastive learning.
What we found
Energy-Efficient Few-Shot Multimodal Transfer
The architecture achieves state-of-the-art performance on benchmark spike datasets and strong few-shot generalization on a new real-world dataset while lowering energy consumption through spiking visual encoding.
Continue the question
Related reading
- SpikeStereoNet: A Brain-Inspired Framework for Stereo Depth Estimation from Spike StreamsICLR 2026 · 2026
- SpikeGrasp: A Benchmark for 6-DoF Grasp Pose Detection from Stereo Spike StreamsarXiv · 2025
Cite this work
Yongchang Gao, Meiling Jin, Zhaofei Yu, Tiejun Huang, Guozhang Chen. SPKLIP: Aligning Spike Video Streams with Natural Language. arXiv, 2025.