Alessandro Moschitti

AAAI Conference 2020 Conference Paper

TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selection

• Siddhant Garg
• Thuy Vu
• Alessandro Moschitti

We propose TANDA, an effective technique for fine-tuning pre-trained Transformer models for natural language tasks. Specifically, we first transfer a pre-trained model into a model for a general task by fine-tuning it with a large and highquality dataset. We then perform a second fine-tuning step to adapt the transferred model to the target domain. We demonstrate the benefits of our approach for answer sentence selection, which is a well-known inference task in Question Answering. We built a large scale dataset to enable the transfer step, exploiting the Natural Questions dataset. Our approach establishes the state of the art on two well-known benchmarks, WikiQA and TREC-QA, achieving the impressive MAP scores of 92% and 94. 3%, respectively, which largely outperform the the highest scores of 83. 4% and 87. 5% of previous work. We empirically show that TANDA generates more stable and robust models reducing the effort required for selecting optimal hyper-parameters. Additionally, we show that the transfer step of TANDA makes the adaptation step more robust to noise. This enables a more effective use of noisy datasets for fine-tuning. Finally, we also confirm the positive impact of TANDA in an industrial setting, using domain specific datasets subject to different types of noise.

AAAI Conference 2019 Conference Paper

Efficient Online Learning for Mapping Kernels on Linguistic Structures

• Giovanni Da San Martino
• Alessandro Sperduti
• Fabio Aiolli
• Alessandro Moschitti

Kernel methods are popular and effective techniques for learning on structured data, such as trees and graphs. One of their major drawbacks is the computational cost related to making a prediction on an example, which manifests in the classification phase for batch kernel methods, and especially in online learning algorithms. In this paper, we analyze how to speed up the prediction when the kernel function is an instance of the Mapping Kernels, a general framework for specifying kernels for structured data which extends the popular convolution kernel framework. We theoretically study the general model, derive various optimization strategies and show how to apply them to popular kernels for structured data. Additionally, we derive a reliable empirical evidence on semantic role labeling task, which is a natural language classification task, highly dependent on syntactic trees. The results show that our faster approach can clearly improve on standard kernel-based SVMs, which cannot run on very large datasets.

AAAI Conference 2019 Conference Paper

Transfer Learning for Sequence Labeling Using Source Model and Target Data

• Lingzhen Chen
• Alessandro Moschitti

In this paper, we propose an approach for transferring the knowledge of a neural model for sequence labeling, learned from the source domain, to a new model trained on a target domain, where new label categories appear. Our transfer learning (TL) techniques enable to adapt the source model using the target data and new categories, without accessing to the source data. Our solution consists in adding new neurons in the output layer of the target model and transferring parameters from the source model, which are then fine-tuned with the target data. Additionally, we propose a neural adapter to learn the difference between the source and the target label distribution, which provides additional important information to the target model. Our experiments on Named Entity Recognition show that (i) the learned knowledge in the source model can be effectively transferred when the target data contains new categories and (ii) our neural adapter further improves such transfer.

JMLR Journal 2018 Journal Article

KELP: a Kernel-based Learning Platform

• Simone Filice
• Giuseppe Castellucci
• Giovanni Da San Martino
• Alessandro Moschitti
• Danilo Croce
• Roberto Basili

KELP is a Java framework that enables fast and easy implementation of kernel functions over discrete data, such as strings, trees or graphs and their combination with standard vectorial kernels. Additionally, it provides several kernel- based algorithms, e.g., online and batch kernel machines for classification, regression and clustering, and a Java environment for easy implementation of new algorithms. KELP is a versatile toolkit, very appealing both to experts and practitioners of machine learning and Java language programming, who can find extensive documentation, tutorials and examples of increasing complexity on the accompanying website. Interestingly, KELP can be also used without any knowledge of Java programming through command line tools and JSON/XML interfaces enabling the declaration and instantiation of articulated learning models using simple templates. Finally, the extensive use of modularity and interfaces in KELP enables developers to easily extend it with their own kernels and algorithms. [abs] [ pdf ][ bib ] [ code ] [ webpage ] &copy JMLR 2018. ( edit, beta )

IJCAI Conference 2013 Conference Paper

Fast Linearization of Tree Kernels over Large-Scale Data

• Aliaksei Severyn
• Alessandro Moschitti

Convolution tree kernels have been successfully applied to many language processing tasks for achieving state-of-the-art accuracy. Unfortunately, higher computational complexity of learning with kernels w. r. t. using explicit feature vectors makes them less attractive for large-scale data. In this paper, we study the latest approaches to solve such problems ranging from feature hashing to reverse kernel engineering and approximate cutting plane training with model compression. We derive a novel method that relies on reverse-kernel engineering together with an efficient kernel learning method. The approach gives the advantage of using tree kernels to automatically generate rich structured feature spaces and working in the linear space where learning and testing is fast. We experimented with training sets up to 4 million examples from Semantic Role Labeling. The results show that (i) the choice of correct structural features is essential and (ii) we can speed-up training from weeks to less than 20 minutes.

ICML Conference 2007 Conference Paper

Fast and effective kernels for relational learning from texts

• Alessandro Moschitti
• Fabio Massimo Zanzotto

ECAI Conference 2006 Conference Paper

Semantic Tree Kernels to Classify Predicate Argument Structures

• Alessandro Moschitti
• Bonaventura Coppola
• Daniele Pighin
• Roberto Basili 0001

Recent work on Semantic Role Labeling (SRL) has shown that syntactic information is critical to detect and extract predicate argument structures. As syntax is expressed by means of structured data, i. e. parse trees, its encoding in learning algorithms is rather complex. In this paper, we apply tree kernels to encode the whole predicate argument structure in Support Vector Machines (SVMs). We extract from the sentence syntactic parse the subtrees that span potential argument structures of the target predicate and classify them in incorrect or correct structures by means of tree kernel based SVMs. Experiments on the PropBank collection show that the classification accuracy of correct/incorrect structures is remarkably high and helps to improve the accuracy of the SRL task. This is a piece of evidence that tree kernels provide a powerful mechanism to learn the complex relation between syntax and semantics.

ECAI Conference 2006 Conference Paper

Shallow Semantic Parsing Based on FrameNet, VerbNet and PropBank

• Ana-Maria Giuglea
• Alessandro Moschitti

This article describes a semantic parser based on FrameNet semantic roles that uses a broad knowledge base created by interconnecting three major resources: FrameNet, VerbNet and PropBank. We link the above resources through a mapping between Intersective Levin classes, which are part of PropBank's annotation, and the FrameNet frames. By using Levin classes, we successfully detect FrameNet semantic roles without relying on the frame information. At the same time, the combined usage of the above resources increases the verb coverage and confers more robustness to our parser. The experiments with Support Vector Machines on automatic Levin class detection suggest that (a) tree kernels are well suited for the task and (b) Intersective Levin classes can be used to improve the accuracy of semantic parsing based on FrameNet roles.

ECAI Conference 2002 Conference Paper

Empirical investigation of fast text classification over linguistic features

• Roberto Basili 0001
• Alessandro Moschitti
• Maria Teresa Pazienza