Author name cluster

Łukasz Kaiser

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

5 papers

1 author row

NeurIPS Conference 2017 Conference Paper

Attention is All you Need

Ashish Vaswani
Noam Shazeer
Niki Parmar
Jakob Uszkoreit
Llion Jones
Aidan Gomez
Łukasz Kaiser
Illia Polosukhin

The dominant sequence transduction models are based on complex recurrent orconvolutional neural networks in an encoder and decoder configuration. The best performing such models also connect the encoder and decoder through an attentionm echanisms. We propose a novel, simple network architecture based solely onan attention mechanism, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superiorin quality while being more parallelizable and requiring significantly less timeto train. Our single model with 165 million parameters, achieves 27. 5 BLEU onEnglish-to-German translation, improving over the existing best ensemble result by over 1 BLEU. On English-to-French translation, we outperform the previoussingle state-of-the-art with model by 0. 7 BLEU, achieving a BLEU score of 41. 1.

PDF Details

NeurIPS Conference 2016 Conference Paper

Can Active Memory Replace Attention?

Łukasz Kaiser
Samy Bengio

Several mechanisms to focus attention of a neural network on selected parts of its input or memory have been used successfully in deep learning models in recent years. Attention has improved image classification, image captioning, speech recognition, generative models, and learning algorithmic tasks, but it had probably the largest impact on neural machine translation. Recently, similar improvements have been obtained using alternative mechanisms that do not focus on a single part of a memory but operate on all of it in parallel, in a uniform way. Such mechanism, which we call active memory, improved over attention in algorithmic tasks, image processing, and in generative modelling. So far, however, active memory has not improved over attention for most natural language processing tasks, in particular for machine translation. We analyze this shortcoming in this paper and propose an extended model of active memory that matches existing attention models on neural machine translation and generalizes better to longer sentences. We investigate this model and explain why previous active memory models did not succeed. Finally, we discuss when active memory brings most benefits and where attention can be a better choice.

PDF Details

NeurIPS Conference 2015 Conference Paper

Grammar as a Foreign Language

Oriol Vinyals
Łukasz Kaiser
Terry Koo
Slav Petrov
Ilya Sutskever
Geoffrey Hinton

Syntactic constituency parsing is a fundamental problem in naturallanguage processing which has been the subject of intensive researchand engineering for decades. As a result, the most accurate parsersare domain specific, complex, and inefficient. In this paper we showthat the domain agnostic attention-enhanced sequence-to-sequence modelachieves state-of-the-art results on the most widely used syntacticconstituency parsing dataset, when trained on a large synthetic corpusthat was annotated using existing parsers. It also matches theperformance of standard parsers when trained on a smallhuman-annotated dataset, which shows that this model is highlydata-efficient, in contrast to sequence-to-sequence models without theattention mechanism. Our parser is also fast, processing over ahundred sentences per second with an unoptimized CPU implementation.

PDF Details

Highlights Conference 2013 Conference Abstract

Machine learning using descriptive complexity and SAT-solvers

Łukasz Kaiser
Charles Jordan

In machine learning there is an inherent trade-off between the efficiency of the learning algorithm and the expressiveness of the hypothesis that one is allowed to learn. Even though this trade-off is present in almost every machine learning task, it is hard to capture formally and still often resolved based on instinct and experience. We propose using parametrized logical formulas to represent hypotheses and then exploit propositional solvers for efficient learning and results from descriptive complexity to get expressiveness guarantees. This allows a more systematic study of the above trade-off and, as we show on a few examples, it is applicable to practical learning tasks. 11: 12 11: 36 Coffee break

PDF Details

TCS Journal 2012 Journal Article

Entanglement and the complexity of directed graphs

Dietmar Berwanger
Erich Grädel
Łukasz Kaiser
Roman Rabinovich

Entanglement is a parameter for the complexity of finite directed graphs that measures to what extent the cycles of the graph are intertwined. It is defined by way of a game similar in spirit to the cops and robber games used to describe treewidth, directed treewidth, and hypertree width. Nevertheless, on many classes of graphs, there are significant differences between entanglement and the various incarnations of treewidth. Entanglement is intimately related with the computational and descriptive complexity of the modal μ -calculus. The number of fixed-point variables needed to describe a finite graph up to bisimulation is captured by its entanglement. This plays a crucial role in the proof that the variable hierarchy of the μ -calculus is strict. We study complexity issues for entanglement and compare it to other structural parameters of directed graphs. One of our main results is that parity games of bounded entanglement can be solved in polynomial time. Specifically, we establish that the complexity of solving a parity game can be parametrised in terms of the minimal entanglement of subgames induced by a winning strategy. Furthermore, we discuss the case of graphs of entanglement two. While graphs of entanglement zero and one are very simple, graphs of entanglement two allow arbitrary nesting of cycles, and they form a sufficiently rich class for modelling relevant classes of structured systems. We provide characterisations of this class, and propose decomposition notions similar to the ones for treewidth, DAG-width, and Kelly-width.

Details DOI