Turning Any Screen Into a Touchless Display with Egocentric Hand Keypoint Detection

Exploring an innovative interaction paradigm, this article discusses using egocentric vision to transform traditional touchscreen interfaces into touchless systems through hand keypoint detection. It details a project that utilizes a point-of-view camera to detect a laptop screen, track hand landmarks in real-time, and convert pinch gestures into zoom actions without physical contact. The system is built using a camera, a laptop, and libraries like MediaPipe and RF-DETR, with Roboflow's InferencePipeline streamlining the computer vision tasks. The architecture is designed to detect the interaction surface, localize the hand, interpret gestures, and execute operating system events, providing a reusable pattern for similar applications. While the current implementation supports basic zoom gestures, future enhancements could include a broader gesture vocabulary, improved lighting handling, and a more robust spatial mapping system to accommodate various camera angles and distances. The project underscores the potential of egocentric vision in creating intuitive, touchless interfaces and invites further exploration of gesture-based interaction frameworks.