Grounding LTLf Specifications in Image Sequences

A critical challenge in neuro-symbolic (NeSy) approaches is to handle the symbol grounding problem without direct supervision. That is mapping high-dimensional raw data into an interpretation over a finite set of abstract concepts with a known meaning, without using labels. In this work, we ground symbols into sequences of images by exploiting symbolic logical knowledge in the form of Linear Temporal Logic over finite traces (LTLf) formulas, and sequence-level labels expressing if a sequence of images is compliant or not with the given formula. Our approach is based on translating the LTLf formula into an equivalent deterministic finite automaton (DFA) and interpreting the latter in fuzzy logic. Experiments show that our system outperforms recurrent neural networks in sequence classification and can reach high image classification accuracy without being trained with any single-image label.

Authors

• Elena Umili
• Roberto Capobianco Sony AI
• Giuseppe De Giacomo University of Oxford

Keywords

• Applications of KR in computer vision
• Grounding representations in the physical world
• Integrating symbolic and sub-symbolic approaches
• Neural-symbolic learning