Author name cluster

Patrice Simard

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

14 papers

1 author row

JMLR Journal 2013 Journal Article

Counterfactual Reasoning and Learning Systems: The Example of Computational Advertising

Léon Bottou
Jonas Peters
Joaquin Quiñonero-Candela
Denis X. Charles
D. Max Chickering
Elon Portugaly
Dipankar Ray
Patrice Simard

This work shows how to leverage causal inference to understand the behavior of complex learning systems interacting with their environment and predict the consequences of changes to the system. Such predictions allow both humans and algorithms to select the changes that would have improved the system performance. This work is illustrated by experiments on the ad placement system associated with the Bing search engine. [abs] [ pdf ][ bib ] &copy JMLR 2013. ( edit, beta )

PDF Details

NeurIPS Conference 2004 Conference Paper

Using Machine Learning to Break Visual Human Interaction Proofs (HIPs)

Kumar Chellapilla
Patrice Simard

Machine learning is often used to automatically solve human tasks. In this paper, we look for tasks where machine learning algorithms are not as good as humans with the hope of gaining insight into their current limitations. We studied various Human Interactive Proofs (HIPs) on the market, because they are systems designed to tell computers and humans apart by posing challenges presumably too hard for computers. We found that most HIPs are pure recognition tasks which can easily be broken using machine learning. The harder HIPs use a combination of segmentation and recognition tasks. From this observation, we found that building segmentation tasks is the most effective way to confuse machine learning algorithms. This has enabled us to build effective HIPs (which we deployed in MSN Passport), as well as design challenging segmentation tasks for machine learning algorithms.

PDF Details

NeurIPS Conference 1998 Conference Paper

Boxlets: A Fast Convolution Algorithm for Signal Processing and Neural Networks

Patrice Simard
Léon Bottou
Patrick Haffner
Yann LeCun

Signal processing and pattern recognition algorithms make exten(cid: 173) sive use of convolution. In many cases, computational accuracy is not as important as computational speed. In feature extraction, for instance, the features of interest in a signal are usually quite distorted. This form of noise justifies some level of quantization in order to achieve faster feature extraction. Our approach consists of approximating regions of the signal with low degree polynomi(cid: 173) als, and then differentiating the resulting signals in order to obtain impulse functions (or derivatives of impulse functions). With this representation, convolution becomes extremely simple and can be implemented quite effectively. The true convolution can be recov(cid: 173) ered by integrating the result of the convolution. This method yields substantial speed up in feature extraction and is applicable to convolutional neural networks.

PDF Details

NeurIPS Conference 1997 Conference Paper

Prior Knowledge in Support Vector Kernels

Bernhard Schölkopf
Patrice Simard
Alex Smola
Vladimir Vapnik

We explore methods for incorporating prior knowledge about a problem at hand in Support Vector learning machines. We show that both invari(cid: 173) ances under group transfonnations and prior knowledge about locality in images can be incorporated by constructing appropriate kernel functions.

PDF Details

NeurIPS Conference 1994 Conference Paper

Learning Prototype Models for Tangent Distance

Trevor Hastie
Patrice Simard

Simard, LeCun & Denker (1993) showed that the performance of nearest-neighbor classification schemes for handwritten character recognition can be improved by incorporating invariance to spe(cid: 173) the so cific transformations in the underlying distance metric - called tangent distance. The resulting classifier, however, can be prohibitively slow and memory intensive due to the large amount of prototypes that need to be stored and used in the distance compar(cid: 173) isons. In this paper we develop rich models for representing large subsets of the prototypes. These models are either used singly per class, or as basic building blocks in conjunction with the K-means clustering algorithm. *This work was performed while Trevor Hastie was a member of the Statistics and Data Analysis Research Group, AT&T Bell Laboratories, Murray Hill, NJ 07974.

PDF Details

NeurIPS Conference 1993 Conference Paper

Backpropagation without Multiplication

Patrice Simard
Hans Graf

The back propagation algorithm has been modified to work with(cid: 173) out any multiplications and to tolerate comput. ations with a low resolution, which makes it. more attractive for a hardware imple(cid: 173) mentatioll. Numbers are represented in float. ing point format with 1 bit mantissa and 3 bits in the exponent for the states, and 1 bit mantissa and 5 bit exponent. for the gradients, while the weights are 16 bit fixed-point numbers. In this way, all the computations can be executed with shift and add operations. Large nehvorks with over 100, 000 weights were t. rained and demonstrat. ed the same per(cid: 173) formance as networks comput. ed with full precision. An estimate of a circuit implementatioll shows that a large network can be placed on a single chip, reaching more t. han 1 billion weight updat. es pel' second. A speedup is also obtained on any machine where a mul(cid: 173) tiplication is slower than a shift operat. ioJl.

PDF Details

NeurIPS Conference 1993 Conference Paper

Efficient Computation of Complex Distance Metrics Using Hierarchical Filtering

Patrice Simard

By their very nature, memory based algorithms such as KNN or Parzen windows require a computationally expensive search of a large database of prototypes. In this paper we optimize the search(cid: 173) ing process for tangent distance (Simard, LeCun and Denker, 1993) to improve speed performance. The closest prototypes are found by recursively searching included subset. s of the database using dis(cid: 173) tances of increasing complexit. y. This is done by using a hierarchy of tangent distances (increasing the Humber of tangent. vectors from o to its maximum) and multiresolution (using wavelets). At each stage, a confidence level of the classification is computed. If the confidence is high enough, the c. omputation of more complex dis(cid: 173) tances is avoided. The resulting algorithm applied to character recognition is close to t. hree orders of magnitude faster than com(cid: 173) puting the full tangent dist. ance on every prot. ot. ypes.

PDF Details

NeurIPS Conference 1992 Conference Paper

Automatic Learning Rate Maximization by On-Line Estimation of the Hessian's Eigenvectors

Yann LeCun
Patrice Simard
Barak Pearlmutter

We propose a very simple, and well principled way of computing the optimal step size in gradient descent algorithms. The on-line version is very efficient computationally, and is applicable to large backpropagation networks trained on large data sets. The main ingredient is a technique for estimating the principal eigenvalue(s) and eigenvector(s) of the objective function's second derivative ma(cid: 173) trix (Hessian), which does not require to even calculate the Hes(cid: 173) sian. Several other applications of this technique are proposed for speeding up learning, or for eliminating useless parameters.

PDF Details

NeurIPS Conference 1992 Conference Paper

Efficient Pattern Recognition Using a New Transformation Distance

Patrice Simard
Yann LeCun
John Denker

Memory-based classification algorithms such as radial basis func(cid: 173) tions or K-nearest neighbors typically rely on simple distances (Eu(cid: 173) clidean, dot product. .. ), which are not particularly meaningful on pattern vectors. More complex, better suited distance measures are often expensive and rather ad-hoc (elastic matching, deformable templates). We propose a new distance measure which (a) can be made locally invariant to any set of transformations of the input and (b) can be computed efficiently. We tested the method on large handwritten character databases provided by the Post Office and the NIST. Using invariances with respect to translation, rota(cid: 173) tion, scaling, shearing and line thickness, the method consistently outperformed all other systems tested on the same databases.

PDF Details

NeurIPS Conference 1992 Conference Paper

Improving Performance in Neural Networks Using a Boosting Algorithm

Harris Drucker
Robert Schapire
Patrice Simard

A boosting algorithm converts a learning machine with error rate less than 50% to one with an arbitrarily low error rate. However, the algorithm discussed here depends on having a large supply of independent training samples. We show how to circumvent this problem and generate an ensemble of learning machines whose performance in optical character recognition problems is dramatically improved over that of a single network. We report the effect of boosting on four databases (all handwritten) consisting of 12, 000 digits from segmented ZIP codes from the United State Postal Service (USPS) and the following from the National Institute of Standards and Testing (NIST): 220, 000 digits, 45, 000 upper case alphas, and 45, 000 lower case alphas. We use two performance measures: the raw error rate (no rejects) and the reject rate required to achieve a 1% error rate on the patterns not rejected. Boosting improved performance in some cases by a factor of three.

PDF Details

NeurIPS Conference 1991 Conference Paper

Reverse TDNN: An Architecture For Trajectory Generation

Patrice Simard
Yann Le Cun

The backpropagation algorithm can be used for both recognition and gen(cid: 173) eration of time trajectories. When used as a recognizer, it has been shown that the performance of a network can be greatly improved by adding structure to the architecture. The same is true in trajectory generation. In particular a new architecture corresponding to a "reversed" TDNN is proposed. Results show dramatic improvement of performance in the gen(cid: 173) eration of hand-written characters. A combination of TDNN and reversed TDNN for compact encoding is also suggested.

PDF Details

NeurIPS Conference 1991 Conference Paper

Tangent Prop - A formalism for specifying selected invariances in an adaptive network

Patrice Simard
Bernard Victorri
Yann LeCun
John Denker

In many machine learning applications, one has access, not only to training data, but also to some high-level a priori knowledge about the desired be(cid: 173) havior of the system. For example, it is known in advance that the output of a character recognizer should be invariant with respect to small spa(cid: 173) tial distortions of the input images (translations, rotations, scale changes, etcetera). We have implemented a scheme that allows a network to learn the deriva(cid: 173) tive of its outputs with respect to distortion operators of our choosing. This not only reduces the learning time and the amount of training data, but also provides a powerful language for specifying what generalizations we wish the network to perform.

PDF Details

NeurIPS Conference 1990 Conference Paper

Shaping the State Space Landscape in Recurrent Networks

Patrice Simard
Jean Raysz
Bernard Victorri

Bernard Victorri ELSAP Universite de Caen 14032 Caen Cedex France Fully recurrent (asymmetrical) networks can be thought of as dynamic systems. The dynamics can be shaped to perform content addressable memories, recognize sequences, or generate trajectories. Unfortunately several problems can arise: First, the convergence in the state space is not guaranteed. Second, the learned fixed points or trajectories are not necessarily stable. Finally, there might exist spurious fixed points and/or spurious "attracting" trajectories that do not correspond to any patterns. In this paper, we introduce a new energy function that presents solutions to all of these problems. We present an efficient gradient descent algorithm which directly acts on the stability of the fixed points and trajectories and on the size and shape of the corresponding basin and valley of attraction. The results are illustrated by the simulation of a small content addressable memory.

PDF Details

NeurIPS Conference 1988 Conference Paper

Fixed Point Analysis for Recurrent Networks

Patrice Simard
Mary Ottaway
Dana Ballard

This paper provides a systematic analysis of the recurrent backpropaga(cid: 173) tion (RBP) algorithm, introducing a number of new results. The main limitation of the RBP algorithm is that it assumes the convergence of the network to a stable fixed point in order to backpropagate the error signals. We show by experiment and eigenvalue analysis that this condi(cid: 173) tion can be violated and that chaotic behavior can be avoided. Next we examine the advantages of RBP over the standard backpropagation al(cid: 173) gorithm. RBP is shown to build stable fixed points corresponding to the input patterns. This makes it an appropriate tool for content address(cid: 173) able memories, one-to-many function learning, and inverse problems.

PDF Details