VLX-Flow: Continuous Video Understanding for Real-Time Multimodal Interaction

VLX-Flow represents a significant advancement in video understanding by enabling continuous, real-time multimodal interaction, addressing the limitations of traditional offline models which process videos only after a query is made. This system processes video streams as sequences of streaming chunks, updating its internal memory incrementally to maintain an evolving visual state, thus allowing it to answer questions from the accumulated context without reprocessing the entire video history. By using Linear Attention and a two-layer memory approach, VLX-Flow ensures stable latency and efficient memory usage, preserving both short-term visual details and long-term semantic context. This supports real-time video question answering and event-triggered interactions, making it particularly valuable for on-device and edge scenarios, where bandwidth, latency, and privacy are concerns. Ultimately, VLX-Flow transforms video understanding into a continuously running perception module, aligning more closely with the persistent observational nature of real-world devices like cameras and robots.