Parsing Embedded Clauses with Distributed Neural Networks

A distributed neural network model called SPEC for processing sentences with recursive relative clauses is described. The model is based on separating the tasks of segmenting the input word sequence into clauses, forming the case-role representations, and keeping track of the recursive embeddings into different modules. The system needs to be trained only with the basic sentence constructs, and it generalizes not only to new instances of familiar relative clause structures, but to novel structures as well. SPEC exhibits plausible memory degradation as the depth of the center embeddings increases, its memory is primed by earlier constituents, and its performance is aided by semantic constraints between the constituents. The ability to process structure is largely due to a central executive network that monitors and controls the execution of the entire system. This way, in contrast to earlier subsymbolic systems, parsing is modeled as a controlled high-level process rather than one based on automatic reflex responses. The girl, who liked the dog, saw the boy’, and it will generalize to different versions of the same structure, such as The dog, who bit the girl, chased the cat (Miikkulainen 1990). However, such a network cannot parse sentences with novel combinations of relative clauses, such as The girl, who liked the dog, saw the boy, who chased the cat. The problem is that distributed neural networks are pattern transformers, and they generalize by interpolating between patterns on which they were trained. They cannot make inferences by dynamically combining processing knowledge that was previously associated to different contexts, such as processing a relative clause at a new place in an otherwise familiar sentence structure. This lack of generalization is a serious problem, given how effortlessly people can understand sentences they have never seen before.

Authors

• Risto Miikkulainen The University of Texas at Austin Cognizant AI Labs

Keywords

No keywords are indexed for this paper.