MemoVision: A Digital Catalog for Everyday Interactions

We present MemoVision, a digital catalog system that captures semantic, spatial, temporal and interaction information as users move around physical environments using client devices such as smart glasses. The system utilizes open-vocabulary semantic segmentation and 3D scans to store objects-of-interest with comprehensive semantic, spatial, temporal and interaction labels. Our demonstration shows multimodal information query and retrieval capabilities, supporting specific queries about object locations, temporal events and user interactions including eye gaze and hand poses, enabling more contextualized responses compared to current multimodal large language models.

Authors

• Lai Xing Ng
• Keith Tien Wei Tang Institute for Infocomm Research, Agency for Science, Technology and Research (A*STAR)
• Jacky Jie Wei Tan Institute for Infocomm Research (I2R), A*STAR

Keywords

No keywords are indexed for this paper.