arXiv · 2025 · Preprint

SPKLIP: Aligning Spike Video Streams with Natural Language

Yongchang Gao, Meiling Jin, Zhaofei Yu, Tiejun Huang, Guozhang Chen

SPKLIP aligns asynchronous spike video streams with natural language through hierarchical multiscale temporal modeling and spike-text contrastive learning.

The scientific question

How can sparse, high-speed spike video streams be aligned with natural language?

Standard vision-language models like CLIP struggle on spike camera outputs because frame-based encoders mismatch sparse, microsecond-resolved event streams.

Hierarchical temporal features extracted from spike streams align with text in a shared space.
Hierarchical temporal features extracted from spike streams align with text in a shared space. Conceptual illustration; not experimental data.

The core idea

Hierarchical Spike-Text Contrastive Alignment

SPKLIP extracts multiscale temporal dynamics from spike streams via a hierarchical encoder (including a full-spiking variant) and aligns visual features with text through contrastive learning.

What we found

Energy-Efficient Few-Shot Multimodal Transfer

The architecture achieves state-of-the-art performance on benchmark spike datasets and strong few-shot generalization on a new real-world dataset while lowering energy consumption through spiking visual encoding.

Continue the question

Related reading

Explore our research

Cite this work

Yongchang Gao, Meiling Jin, Zhaofei Yu, Tiejun Huang, Guozhang Chen. SPKLIP: Aligning Spike Video Streams with Natural Language. arXiv, 2025.