Antoine Bordes

AAAI Conference 2018 Conference Paper

StarSpace: Embed All The Things!

• Ledell Wu
• Adam Fisch
• Sumit Chopra
• Keith Adams
• Antoine Bordes
• Jason Weston

We present StarSpace, a general-purpose neural embedding model that can solve a wide variety of problems: labeling tasks such as text classification, ranking tasks such as information retrieval/web search, collaborative filtering-based or content-based recommendation, embedding of multirelational graphs, and learning word, sentence or document level embeddings. In each case the model works by embedding those entities comprised of discrete features and comparing them against each other – learning similarities dependent on the task. Empirical results on a number of tasks show that StarSpace is highly competitive with existing methods, whilst also being generally applicable to new cases where those methods are not.

NeurIPS Conference 2017 Conference Paper

Fader Networks:Manipulating Images by Sliding Attributes

• Guillaume Lample
• Neil Zeghidour
• Nicolas Usunier
• Antoine Bordes
• Ludovic Denoyer
• Marc'Aurelio Ranzato

This paper introduces a new encoder-decoder architecture that is trained to reconstruct images by disentangling the salient information of the image and the values of attributes directly in the latent space. As a result, after training, our model can generate different realistic versions of an input image by varying the attribute values. By using continuous attribute values, we can choose how much a specific attribute is perceivable in the generated image. This property could allow for applications where users can modify an image using sliding knobs, like faders on a mixing console, to change the facial expression of a portrait, or to update the color of some objects. Compared to the state-of-the-art which mostly relies on training adversarial networks in pixel space by altering attribute values at train time, our approach results in much simpler training schemes and nicely scales to multiple attributes. We present evidence that our model can significantly change the perceived value of the attributes while preserving the naturalness of images.

ICLR Conference 2017 Conference Paper

Learning End-to-End Goal-Oriented Dialog

• Antoine Bordes
• Y-Lan Boureau
• Jason Weston

Traditional dialog systems used in goal-oriented applications require a lot of domain-specific handcrafting, which hinders scaling up to new domains. End- to-end dialog systems, in which all components are trained from the dialogs themselves, escape this limitation. But the encouraging success recently obtained in chit-chat dialog may not carry over to goal-oriented settings. This paper proposes a testbed to break down the strengths and shortcomings of end-to-end dialog systems in goal-oriented applications. Set in the context of restaurant reservation, our tasks require manipulating sentences and symbols, so as to properly conduct conversations, issue API calls and use the outputs of such calls. We show that an end-to-end dialog system based on Memory Networks can reach promising, yet imperfect, performance and learn to perform non-trivial operations. We confirm those results by comparing our system to a hand-crafted slot-filling baseline on data from the second Dialog State Tracking Challenge (Henderson et al., 2014a). We show similar result patterns on data extracted from an online concierge service.

JAIR Journal 2016 Journal Article

Combining Two and Three-Way Embedding Models for Link Prediction in Knowledge Bases

• Alberto Garcia-Duran
• Antoine Bordes
• Nicolas Usunier
• Yves Grandvalet

This paper tackles the problem of endogenous link prediction for knowledge base completion. Knowledge bases can be represented as directed graphs whose nodes correspond to entities and edges to relationships. Previous attempts either consist of powerful systems with high capacity to model complex connectivity patterns, which unfortunately usually end up overfitting on rare relationships, or in approaches that trade capacity for simplicity in order to fairly model all relationships, frequent or not. In this paper, we propose Tatec, a happy medium obtained by complementing a high-capacity model with a simpler one, both pre-trained separately and then combined. We present several variants of this model with different kinds of regularization and combination strategies and show that this approach outperforms existing methods on different types of relationships by achieving state-of-the-art results on four benchmarks of the literature.

ICLR Conference 2016 Conference Paper

The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations

• Felix Hill
• Antoine Bordes
• Sumit Chopra
• Jason Weston

We introduce a new test of how well language models capture meaning in children's books. Unlike standard language modelling benchmarks, it distinguishes the task of predicting syntactic function words from that of predicting lower-frequency words, which carry greater semantic content. We compare a range of state-of-the-art models, each with a different way of encoding what has been previously read. We show that models which store explicit representations of long-term contexts outperform state-of-the-art neural language models at predicting semantic content words, although this advantage is not observed for syntactic function words. Interestingly, we find that the amount of text encoded in a single memory representation is highly influential to the performance: there is a sweet-spot, not too big and not too small, between single words and full sentences that allows the most meaningful information in a text to be effectively retained and recalled. Further, the attention over such window-based memories can be trained effectively through self-supervision. We then assess the generality of this principle by applying it to the CNN QA benchmark, which involves identifying named entities in paraphrased summaries of news articles, and achieve state-of-the-art performance.

ICLR Conference 2015 Conference Paper

Memory Networks

• Jason Weston
• Sumit Chopra
• Antoine Bordes

We describe a new class of learning models called memory networks. Memory networks reason with inference components combined with a long-term memory component; they learn how to use these jointly. The long-term memory can be read and written to, with the goal of using it for prediction. We investigate these models in the context of question answering (QA) where the long-term memory effectively acts as a (dynamic) knowledge base, and the output is a textual response. We evaluate them on a large-scale QA task, and a smaller, but more complex, toy task generated from a simulated world. In the latter, we show the reasoning power of such models by chaining multiple supporting sentences to answer questions that require understanding the intension of verbs.

NeurIPS Conference 2013 Conference Paper

Translating Embeddings for Modeling Multi-relational Data

• Antoine Bordes
• Nicolas Usunier
• Alberto Garcia-Duran
• Jason Weston
• Oksana Yakhnenko

We consider the problem of embedding entities and relationships of multi-relational data in low-dimensional vector spaces. Our objective is to propose a canonical model which is easy to train, contains a reduced number of parameters and can scale up to very large databases. Hence, we propose, TransE, a method which models relationships by interpreting them as translations operating on the low-dimensional embeddings of the entities. Despite its simplicity, this assumption proves to be powerful since extensive experiments show that TransE significantly outperforms state-of-the-art methods in link prediction on two knowledge bases. Besides, it can be successfully trained on a large scale data set with 1M entities, 25k relationships and more than 17M training samples.

NeurIPS Conference 2012 Conference Paper

A latent factor model for highly multi-relational data

• Rodolphe Jenatton
• Nicolas Roux
• Antoine Bordes
• Guillaume Obozinski

Many data such as social networks, movie preferences or knowledge bases are multi-relational, in that they describe multiple relationships between entities. While there is a large body of work focused on modeling these data, few considered modeling these multiple types of relationships jointly. Further, existing approaches tend to breakdown when the number of these types grows. In this paper, we propose a method for modeling large multi-relational datasets, with possibly thousands of relations. Our model is based on a bilinear structure, which captures the various orders of interaction of the data, but also shares sparse latent factors across different relations. We illustrate the performance of our approach on standard tensor-factorization datasets where we attain, or outperform, state-of-the-art results. Finally, a NLP application demonstrates our scalability and the ability of our model to learn efficient, and semantically meaningful verb representations.

ICML Conference 2011 Conference Paper

Domain Adaptation for Large-Scale Sentiment Classification: A Deep Learning Approach

• Xavier Glorot
• Antoine Bordes
• Yoshua Bengio

AAAI Conference 2011 Conference Paper

Learning Structured Embeddings of Knowledge Bases

• Antoine Bordes
• Jason Weston
• Ronan Collobert
• Yoshua Bengio

Many Knowledge Bases (KBs) are now readily available and encompass colossal quantities of information thanks to either a long-term funding effort (e. g. WordNet, OpenCyc) or a collaborative process (e. g. Freebase, DBpedia). However, each of them is based on a different rigid symbolic framework which makes it hard to use their data in other systems. It is unfortunate because such rich structured knowledge might lead to a huge leap forward in many other areas of AI like natural language processing (word-sense disambiguation, natural language understanding, .. .), vision (scene classification, image semantic annotation, .. .) or collaborative filtering. In this paper, we present a learning process based on an innovative neural network architecture designed to embed any of these symbolic representations into a more flexible continuous vector space in which the original knowledge is kept and enhanced. These learnt embeddings would allow data from any KB to be easily used in recent machine learning methods for prediction and information retrieval. We illustrate our method on WordNet and Freebase and also present a way to adapt it to knowledge extraction from raw text.

JMLR Journal 2010 Journal Article

Erratum: SGDQN is Less Careful than Expected

• Antoine Bordes
• Léon Bottou
• Patrick Gallinari
• Jonathan Chang
• S. Alex Smith

The SGD-QN algorithm described in Bordes et al. (2009) contains a subtle flaw that prevents it from reaching its design goals. Yet the flawed SGD-QN algorithm has worked well enough to be a winner of the first Pascal Large Scale Learning Challenge (Sonnenburg et al., 2008). This document clarifies the situation, proposes a corrected algorithm, and evaluates its performance. [abs] [ pdf ][ bib ] &copy JMLR 2010. ( edit, beta )

ICML Conference 2010 Conference Paper

Label Ranking under Ambiguous Supervision for Learning Semantic Correspondences

• Antoine Bordes
• Nicolas Usunier
• Jason Weston

JMLR Journal 2009 Journal Article

SGD-QN: Careful Quasi-Newton Stochastic Gradient Descent

• Antoine Bordes
• Léon Bottou
• Patrick Gallinari

The SGD-QN algorithm is a stochastic gradient descent algorithm that makes careful use of second-order information and splits the parameter update into independently scheduled components. Thanks to this design, SGD-QN iterates nearly as fast as a first-order stochastic gradient descent but requires less iterations to achieve the same accuracy. This algorithm won the "Wild Track" of the first PASCAL Large Scale Learning Challenge (Sonnenburg et al., 2008). [abs] [ pdf ][ bib ] &copy JMLR 2009. ( edit, beta )

ICML Conference 2007 Conference Paper

Solving multiclass support vector machines with LaRank

• Antoine Bordes
• Léon Bottou
• Patrick Gallinari
• Jason Weston

JMLR Journal 2005 Journal Article

Fast Kernel Classifiers with Online and Active Learning

• Antoine Bordes
• Seyda Ertekin
• Jason Weston
• Léon Bottou

Very high dimensional learning systems become theoretically possible when training examples are abundant. The computing cost then becomes the limiting factor. Any efficient learning algorithm should at least take a brief look at each example. But should all examples be given equal attention? This contribution proposes an empirical answer. We first present an online SVM algorithm based on this premise. LASVM yields competitive misclassification rates after a single pass over the training examples, outspeeding state-of-the-art SVM solvers. Then we show how active example selection can yield faster training, higher accuracies, and simpler models, using only a fraction of the training example labels. [abs] [ pdf ][ bib ] &copy JMLR 2005. ( edit, beta )