Author name cluster

Benjamin Schrauwen

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

14 papers

2 author rows

NeurIPS Conference 2014 Conference Paper

Factoring Variations in Natural Images with Deep Gaussian Mixture Models

Aaron van den Oord
Benjamin Schrauwen

Generative models can be seen as the swiss army knives of machine learning, as many problems can be written probabilistically in terms of the distribution of the data, including prediction, reconstruction, imputation and simulation. One of the most promising directions for unsupervised learning may lie in Deep Learning methods, given their success in supervised learning. However, one of the current problems with deep unsupervised learning methods, is that they often are harder to scale. As a result there are some easier, more scalable shallow methods, such as the Gaussian Mixture Model and the Student-t Mixture Model, that remain surprisingly competitive. In this paper we propose a new scalable deep generative model for images, called the Deep Gaussian Mixture Model, that is a straightforward but powerful generalization of GMMs to multiple layers. The parametrization of a Deep GMM allows it to efficiently capture products of variations in natural images. We propose a new EM-based algorithm that scales well to large datasets, and we show that both the Expectation and the Maximization steps can easily be distributed over multiple machines. In our density estimation experiments we show that deeper GMM architectures generalize better than more shallow ones, with results in the same ballpark as the state of the art.

PDF Details

JMLR Journal 2014 Journal Article

The Student-t Mixture as a Natural Image Patch Prior with Application to Image Compression

Aäron van den Oord
Benjamin Schrauwen

Recent results have shown that Gaussian mixture models (GMMs) are remarkably good at density modeling of natural image patches, especially given their simplicity. In terms of log likelihood on real-valued data they are comparable with the best performing techniques published, easily outperforming more advanced ones, such as deep belief networks. They can be applied to various image processing tasks, such as image denoising, deblurring and inpainting, where they improve on other generic prior methods, such as sparse coding and field of experts. Based on this we propose the use of another, even richer mixture model based image prior: the Student-t mixture model (STM). We demonstrate that it convincingly surpasses GMMs in terms of log likelihood, achieving performance competitive with the state of the art in image patch modeling. We apply both the GMM and STM to the task of lossy and lossless image compression, and propose efficient coding schemes that can easily be extended to other unsupervised machine learning models. Finally, we show that the suggested techniques outperform JPEG, with results comparable to or better than JPEG 2000. [abs] [ pdf ][ bib ] &copy JMLR 2014. ( edit, beta )

PDF Details

NeurIPS Conference 2013 Conference Paper

Deep content-based music recommendation

Aaron van den Oord
Sander Dieleman
Benjamin Schrauwen

Automatic music recommendation has become an increasingly relevant problem in recent years, since a lot of music is now sold and consumed digitally. Most recommender systems rely on collaborative filtering. However, this approach suffers from the cold start problem: it fails when no usage data is available, so it is not effective for recommending new and unpopular songs. In this paper, we propose to use a latent factor model for recommendation, and predict the latent factors from music audio when they cannot be obtained from usage data. We compare a traditional approach using a bag-of-words representation of the audio signals with deep convolutional neural networks, and evaluate the predictions quantitatively and qualitatively on the Million Song Dataset. We show that using predicted latent factors produces sensible recommendations, despite the fact that there is a large semantic gap between the characteristics of a song that affect user preference and the corresponding audio signal. We also show that recent advances in deep learning translate very well to the music recommendation setting, with deep convolutional neural networks significantly outperforming the traditional approach.

PDF Details

EAAI Journal 2013 Journal Article

Time series classification for the prediction of dialysis in critically ill patients using echo statenetworks

Femke Ongenae
Stijn Van Looy
David Verstraeten
Thierry Verplancke
Dominique Benoit
Filip De Turck
Tom Dhaene
Benjamin Schrauwen

Objective: Time series often appear in medical databases, but only few machine learning methods exist that process this kind of data properly. Most modeling techniques have been designed with a static data model in mind and are not suitable for coping with the dynamic nature of time series. Recurrent neural networks (RNNs) are often used to process time series, but only a few training algorithms exist for RNNs which are complex and often yield poor results. Therefore, researchers often turn to traditional machine learning approaches, such as support vector machines (SVMs), which can easily be set up and trained and combine them with feature extraction (FE) and selection (FS) to process the high-dimensional temporal data. Recently, a new approach, called echo state networks (ESNs), has been developed to simplify the training process of RNNs. This approach allows modeling the dynamics of a system based on time series data in a straightforwardway. The objective of this study is to explore the advantages of using ESN instead of other traditional classifiers combined with FE and FS in classification problems in the intensive care unit (ICU) when the input data consists of time series. While ESNs have mostly been used to predict the future course of a time series, we use the ESN model for classification instead. Although time series often appear in medical data, little medical applications of ESNs have been studiedyet. Methods and material: ESN is used to predict the need for dialysis between the fifth and tenth day after admission in the ICU. The input time series consist of measured diuresis and creatinine values during the first 3days after admission. Data about 830 patients was used for the study, of which 82 needed dialysis between the fifth and tenth day after admission. ESN is compared to traditional classifiers, a sophisticated and a simple one, namely support vector machines and the naive Bayes (NB) classifier. Prior to the use of the SVM and NB classifier, FE and FS is required to reduce the number of input features and thus alleviate the curse dimensionality. Extensive feature extraction was applied to capture both the overall properties of the time series and the correlation between the different measurements in the time series. The feature selection method consists of a greedy hybrid filter-wrapper method using a NB classifier, which selects in each iteration the feature that improves prediction the best and shows little multicollinearity with the already selected set. Least squares regression with noise was used to train the linear readout function of the ESN to mitigate sensitivity to noise and overfitting. Fisher labeling was used to deal with the unbalanced data set. Parameter sweeps were performed to determine the optimal parameter values for the different classifiers. The area under the curve (AUC) and maximum balanced accuracy are used as performance measures. The required execution time was also measured. Results: The classification performance of the ESN shows significant difference at the 5% level compared to the performance of the SVM or the NB classifier combined with FE and FS. The NB+FE+FS, with an average AUC of 0. 874, has the best classification performance. This classifier is followed by the ESN, which has an average AUC of 0. 849. The SVM+FE+FS has the worst performance with an average AUC of 0. 838. The computation time needed to pre-process the data and to train and test the classifier is significantly less for the ESN compared to the SVM andNB. Conclusion: It can be concluded that the use of ESN has an added value in predicting the need for dialysis through the analysis of time series data. The ESN requires significantly less processing time, needs no domain knowledge, is easy to implement, and can be configured using rules ofthumb.

Details DOI

NeurIPS Conference 2013 Conference Paper

Training and Analysing Deep Recurrent Neural Networks

Michiel Hermans
Benjamin Schrauwen

Time series often have a temporal hierarchy, with information that is spread out over multiple time scales. Common recurrent neural networks, however, do not explicitly accommodate such a hierarchy, and most research on them has been focusing on training algorithms rather than on their basic architecture. In this pa- per we study the effect of a hierarchy of recurrent neural networks on processing time series. Here, each layer is a recurrent network which receives the hidden state of the previous layer as input. This architecture allows us to perform hi- erarchical processing on difficult temporal tasks, and more naturally capture the structure of time series. We show that they reach state-of-the-art performance for recurrent networks in character-level language modelling when trained with sim- ple stochastic gradient descent. We also offer an analysis of the different emergent time scales.

PDF Details

JMLR Journal 2013 Journal Article

Training Energy-Based Models for Time-Series Imputation

Philémon Brakel
Dirk Stroobandt
Benjamin Schrauwen

Imputing missing values in high dimensional time-series is a difficult problem. This paper presents a strategy for training energy-based graphical models for imputation directly, bypassing difficulties probabilistic approaches would face. The training strategy is inspired by recent work on optimization-based learning (Domke, 2012) and allows complex neural models with convolutional and recurrent structures to be trained for imputation tasks. In this work, we use this training strategy to derive learning rules for three substantially different neural architectures. Inference in these models is done by either truncated gradient descent or variational mean-field iterations. In our experiments, we found that the training methods outperform the Contrastive Divergence learning algorithm. Moreover, the training methods can easily handle missing values in the training data itself during learning. We demonstrate the performance of this learning scheme and the three models we introduce on one artificial and two real-world data sets. [abs] [ pdf ][ bib ] &copy JMLR 2013. ( edit, beta )

PDF Details

NeurIPS Conference 2012 Conference Paper

A P300 BCI for the Masses: Prior Information Enables Instant Unsupervised Spelling

Pieter-Jan Kindermans
Hannes Verschore
David Verstraeten
Benjamin Schrauwen

The usability of Brain Computer Interfaces (BCI) based on the P300 speller is severely hindered by the need for long training times and many repetitions of the same stimulus. In this contribution we introduce a set of unsupervised hierarchical probabilistic models that tackle both problems simultaneously by incorporating prior knowledge from two sources: information from other training subjects (through transfer learning) and information about the words being spelled (through language models). We show, that due to this prior knowledge, the performance of the unsupervised models parallels and in some cases even surpasses that of supervised models, while eliminating the tedious training session.

PDF Details

NeurIPS Conference 2012 Conference Paper

Analog readout for optical reservoir computers

Anteo Smerieri
François Duport
Yvon Paquot
Benjamin Schrauwen
Marc Haelterman
Serge Massar

Reservoir computing is a new, powerful and flexible machine learning technique that is easily implemented in hardware. Recently, by using a time-multiplexed architecture, hardware reservoir computers have reached performance comparable to digital implementations. Operating speeds allowing for real time information operation have been reached using optoelectronic systems. At present the main performance bottleneck is the readout layer which uses slow, digital postprocessing. We have designed an analog readout suitable for time-multiplexed optoelectronic reservoir computers, capable of working in real time. The readout has been built and tested experimentally on a standard benchmark task. Its performance is better than non-reservoir methods, with ample room for further improvement. The present work thereby overcomes one of the major limitations for the future development of hardware reservoir computers.

PDF Details

JMLR Journal 2012 Journal Article

Oger: Modular Learning Architectures For Large-Scale Sequential Processing

David Verstraeten
Benjamin Schrauwen
Sander Dieleman
Philemon Brakel
Pieter Buteneers
Dejan Pecevski

Oger (OrGanic Environment for Reservoir computing) is a Python toolbox for building, training and evaluating modular learning architectures on large data sets. It builds on MDP for its modularity, and adds processing of sequential data sets, gradient descent training, several cross-validation schemes and parallel parameter optimization methods. Additionally, several learning algorithms are implemented, such as different reservoir implementations (both sigmoid and spiking), ridge regression, conditional restricted Boltzmann machine (CRBM) and others, including GPU accelerated versions. Oger is released under the GNU LGPL, and is available from http://organic.elis.ugent.be/oger. [abs] [ pdf ][ bib ] [ code ] &copy JMLR 2012. ( edit, beta )

PDF Details

AIIM Journal 2011 Journal Article

Automatic detection of epileptic seizures on the intra-cranial electroencephalogram of rats using reservoir computing

Pieter Buteneers
David Verstraeten
Pieter van Mierlo
Tine Wyckhuys
Dirk Stroobandt
Robrecht Raedt
Hans Hallez
Benjamin Schrauwen

Introduction In this paper we propose a technique based on reservoir computing (RC) to mark epileptic seizures on the intra-cranial electroencephalogram (EEG) of rats. RC is a recurrent neural networks training technique which has been shown to possess good generalization properties with limited training. Materials The system is evaluated on data containing two different seizure types: absence seizures from genetic absence epilepsy rats from Strasbourg (GAERS) and tonic–clonic seizures from kainate-induced temporal-lobe epilepsy rats. The dataset consists of 452hours from 23 GAERS and 982hours from 15 kainate-induced temporal-lobe epilepsy rats. Methods During the preprocessing stage, several features are extracted from the EEG. A feature selection algorithm selects the best features, which are then presented as input to the RC-based classification algorithm. To classify the output of this algorithm a two-threshold technique is used. This technique is compared with other state-of-the-art techniques. Results A balanced error rate (BER) of 3. 7% and 3. 5% was achieved on the data from GAERS and kainate rats, respectively. This resulted in a sensitivity of 96% and 94% and a specificity of 96% and 99% respectively. The state-of-the-art technique for GAERS achieved a BER of 4%, whereas the best technique to detect tonic–clonic seizures achieved a BER of 16%. Conclusion Our method outperforms up-to-date techniques and only a few parameters need to be optimized on a limited training set. It is therefore suited as an automatic aid for epilepsy researchers and is able to eliminate the tedious manual review and annotation of EEG.

Details DOI

NeurIPS Conference 2010 Conference Paper

Phoneme Recognition with Large Hierarchical Reservoirs

Fabian Triefenbach
Azarakhsh Jalalvand
Benjamin Schrauwen
Jean-pierre Martens

Automatic speech recognition has gradually improved over the years, but the reliable recognition of unconstrained speech is still not within reach. In order to achieve a breakthrough, many research groups are now investigating new methodologies that have potential to outperform the Hidden Markov Model technology that is at the core of all present commercial systems. In this paper, it is shown that the recently introduced concept of Reservoir Computing might form the basis of such a methodology. In a limited amount of time, a reservoir system that can recognize the elementary sounds of continuous speech has been built. The system already achieves a state-of-the-art performance, and there is evidence that the margin for further improvements is still significant.

PDF Details

ICRA Conference 2010 Conference Paper

Supervised learning of internal models for autonomous goal-oriented robot navigation using Reservoir Computing

Eric A. Antonelo
Benjamin Schrauwen

In this work we propose a hierarchical architecture which constructs internal models of a robot environment for goal-oriented navigation by an imitation learning process. The proposed architecture is based on the Reservoir Computing paradigm for training Recurrent Neural Networks (RNN). It is composed of two randomly generated RNNs (called reservoirs), one for modeling the localization capability and one for learning the navigation skill. The localization module is trained to detect the current and previously visited robot rooms based only on 8 noisy infra-red distance sensors. These predictions together with distance sensors and the desired goal location are used by the navigation network to actually steer the robot through the environment in a goal-oriented manner. The training of this architecture is performed in a supervised way (with examples of trajectories created by a supervisor) using linear regression on the reservoir states. So, the reservoir acts as a temporal kernel projecting the inputs to a rich feature space, whose states are linearly combined to generate the desired outputs. Experimental results on a simulated robot show that the trained system can localize itself within both simple and large unknown environments and navigate successfully to desired goals.

Details

ICRA Conference 2008 Conference Paper

Mobile robot control in the road sign problem using Reservoir Computing networks

Eric A. Antonelo
Benjamin Schrauwen
Dirk Stroobandt

In this work we tackle the road sign problem with reservoir computing (RC) networks. The T-maze task (a particular form of the road sign problem) consists of a robot in a T-shaped environment that must reach the correct goal (left or right arm of the T-maze) depending on a previously received input sign. It is a control task in which the delay period between the sign received and the required response (e. g. , turn right or left) is a crucial factor. Delayed response tasks like this one form a temporal problem that can be handled very well by RC networks. Reservoir computing is a biologically plausible technique which overcomes the problems of previous algorithms such as backpropagation through time - which exhibits slow (or non-) convergence on training. RC is a new concept that includes a fast and efficient training algorithm. We show that this simple approach can solve the T-maze task efficiently.

Details

NeurIPS Conference 2008 Conference Paper

On Computational Power and the Order-Chaos Phase Transition in Reservoir Computing

Benjamin Schrauwen
Lars Buesing
Robert Legenstein

Randomly connected recurrent neural circuits have proven t o be very powerful models for online computations when a trained memoryless re adout function is appended. Such Reservoir Computing (RC) systems are commonly used in two flavors: with analog or binary (spiking) neurons in the recur rent circuits. Previous work showed a fundamental difference between these two incarnations of the RC idea. The performance of a RC system built from binary neuron s seems to depend strongly on the network connectivity structure. In network s of analog neurons such dependency has not been observed. In this article we investigate this apparent dichotomy in terms of the in-degree of the circuit nodes. Our analyses based amongst others on the Lyapunov exponent reveal that the phase transition between ordered and chaotic network behavior of binary circuits qua litatively differs from the one in analog circuits. This explains the observed decre ased computational performance of binary circuits of high node in-degree. Furt hermore, a novel mean-field predictor for computational performance is intr oduced and shown to accurately predict the numerically obtained results.

PDF Details