Generating Coordinated Natural Language and 3D Animations for Complex Spatial Explanations

Dynamically providingstudents with clear explanationsof complex spatial conceptsis critical for a broad range of knowledge-based educational and training systems. Thiscalls for a realtimesolution that can dynamicallycreate 3Danimatedexplanations that artfully integrate well-chosen speech with rich visualizations. Unfortunately, planning the integrated creation of 3Danimationand spatial linguistic utterancesin realtimerequires coordinating the visual presentation of 3Dobjects and generating appropriate spatial phrases that accuratelyreflect the relative position, orientation, and direction of the objects presented. We present a visuo-linguistic framework for generating multimediaspatial explanationscombining 3D animation and speech that complement one another. Because 3Danimation planners require spatial knowledge in a geometricform and natural languagegenerators require spatial knowledgein a linguistic form, a realtime multimedia plannerinterposedbetween the visual and linguistic componentscan serve as a mediator. This frameworkhas been implementedin CINESPEAK, a multimediaexplanationgenerator consisting of a visuo-linguistic mediator, a 3Danimationplanner, and a realtime natural language generator with a speech synthesizer. CINESPEAK has been used in conjunctionwith a prototype 3Dlearning environment in the domain of physics to generate realtime multimedia explanationsof three dimensionalelectromagnetic fields, forces, andelectrical current.

Authors

• Stuart G. Towns Charles B. Callaway

Keywords

No keywords are indexed for this paper.