Shimon Ullman

NeurIPS Conference 2024 Conference Paper

Biologically Inspired Learning Model for Instructed Vision

• Roy Abel
• Shimon Ullman

As part of the effort to understand how the brain learns, ongoing research seeks to combine biological knowledge with current artificial intelligence (AI) modeling in an attempt to find an efficient biologically plausible learning scheme. Current models often use a cortical-like combination of bottom-up (BU) and top-down (TD) processing, where the TD part carries feedback signals for learning. However, in the visual cortex, the TD pathway plays a second major role in visual attention, by guiding the visual process toward locations and tasks of interest. A biological model should therefore integrate both learning and visual guidance. We introduce a model that uses a cortical-like combination of BU and TD processing that naturally integrates the two major functions of the TD stream. This integration is achieved through an appropriate connectivity pattern between the BU and TD streams, a novel processing cycle that uses the TD stream twice, and a 'Counter-Hebb' learning mechanism that operates across both streams. We show that the 'Counter-Hebb' mechanism can provide an exact backpropagation synaptic modification. Additionally, our model can effectively guide the visual stream to perform a task of interest, achieving competitive performance on standard multi-task learning benchmarks compared to AI models. The successful combination of learning and visual guidance could provide a new view on combining BU and TD processing in human vision and suggests possible directions for both biologically plausible models and artificial instructed models, such as vision-language models (VLMs).

NeurIPS Conference 2023 Conference Paper

Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Models

• Sivan Doveh
• Assaf Arbelle
• Sivan Harary
• Roei Herzig
• Donghyun Kim
• Paola Cascante-Bonilla
• Amit Alfassy
• Rameswar Panda

Vision and Language (VL) models offer an effective method for aligning representation spaces of images and text allowing for numerous applications such as cross-modal retrieval, visual and multi-hop question answering, captioning, and many more. However, the aligned image-text spaces learned by all the popular VL models are still suffering from the so-called 'object bias' - their representations behave as 'bags of nouns' mostly ignoring or downsizing the attributes, relations, and states of objects described/appearing in texts/images. Although some great attempts at fixing these `compositional reasoning' issues were proposed in the recent literature, the problem is still far from being solved. In this paper, we uncover two factors limiting the VL models' compositional reasoning performance. These two factors are properties of the paired VL dataset used for finetuning (or pre-training) the VL model: (i) the caption quality, or in other words 'image-alignment', of the texts; and (ii) the 'density' of the captions in the sense of mentioning all the details appearing on the image. We propose a fine-tuning approach for automatically treating these factors on a standard collection of paired VL data (CC3M). Applied to CLIP, we demonstrate its significant compositional reasoning performance increase of up to $\sim27$\% over the base model, up to $\sim20$\% over the strongest baseline, and by $6. 7$\% on average. Our code is provided in the Supplementary and would be released upon acceptance.

AAAI Conference 2020 Conference Paper

Cakewalk Sampling

• Uri Patish
• Shimon Ullman

We study the task of finding good local optima in combinatorial optimization problems. Although combinatorial optimization is NP-hard in general, locally optimal solutions are frequently used in practice. Local search methods however typically converge to a limited set of optima that depend on their initialization. Sampling methods on the other hand can access any valid solution, and thus can be used either directly or alongside methods of the former type as a way for finding good local optima. Since the effectiveness of this strategy depends on the sampling distribution, we derive a robust learning algorithm that adapts sampling distributions towards good local optima of arbitrary objective functions. As a first use case, we empirically study the efficiency in which sampling methods can recover locally maximal cliques in undirected graphs. Not only do we show how our adaptive sampler outperforms related methods, we also show how it can even approach the performance of established clique algorithms. As a second use case, we consider how greedy algorithms can be combined with our adaptive sampler, and we demonstrate how this leads to superior performance in k-medoid clustering. Together, these findings suggest that our adaptive sampler can provide an effective strategy to combinatorial optimization problems that arise in practice.

ICML Conference 2015 Conference Paper

Learning Local Invariant Mahalanobis Distances

• Ethan Fetaya
• Shimon Ullman

For many tasks and data types, there are natural transformations to which the data should be invariant or insensitive. For instance, in visual recognition, natural images should be insensitive to rotation and translation. This requirement and its implications have been important in many machine learning applications, and tolerance for image transformations was primarily achieved by using robust feature vectors. In this paper we propose a novel and computationally efficient way to learn a local Mahalanobis metric per datum, and show how we can learn a local invariant metric to any transformation in order to improve performance.

NeurIPS Conference 2010 Conference Paper

Using body-anchored priors for identifying actions in single images

• Leonid Karlinsky
• Michael Dinerstein
• Shimon Ullman

This paper presents an approach to the visual recognition of human actions using only single images as input. The task is easy for humans but difficult for current approaches to object recognition, because action instances may be similar in terms of body pose, and often require detailed examination of relations between participating objects and body parts in order to be recognized. The proposed approach applies a two-stage interpretation procedure to each training and test image. The first stage produces accurate detection of the relevant body parts of the actor, forming a prior for the local evidence needed to be considered for identifying the action. The second stage extracts features that are ‘anchored’ to the detected body parts, and uses these features and their feature-to-part relations in order to recognize the action. The body anchored priors we propose apply to a large range of human actions. These priors allow focusing on the relevant regions and relations, thereby significantly simplifying the learning process and increasing recognition performance.

ICML Conference 2007 Conference Paper

Uncovering shared structures in multiclass classification

• Yonatan Amit
• Michael Fink 0002
• Nathan Srebro
• Shimon Ullman

This paper suggests a method for multiclass learning with many classes by simultaneously learning shared characteristics common to the classes, and predictors for the classes in terms of these characteristics. We cast this as a convex optimization problem, using trace-norm regularization and study gradient-based optimization both for the linear case and the kernelized setting.

ICML Conference 2006 Conference Paper

Online multiclass learning by interclass hypothesis sharing

• Michael Fink 0002
• Shai Shalev-Shwartz
• Yoram Singer
• Shimon Ullman

NeurIPS Conference 1991 Conference Paper

Linear Operator for Object Recognition

• Ronen Basri
• Shimon Ullman

Visual object recognition involves the identification of images of 3-D ob(cid: 173) jects seen from arbitrary viewpoints. We suggest an approach to object recognition in which a view is represented as a collection of points given by their location in the image. An object is modeled by a set of 2-D views together with the correspondence between the views. We show that any novel view of the object can be expressed as a linear combination of the stored views. Consequently, we build a linear operator that distinguishes between views of a specific object and views of other objects. This opera(cid: 173) tor can be implemented using neural network architectures with relatively simple structures.

NeurIPS Conference 1990 Conference Paper

Grouping Contours by Iterated Pairing Network

• Amnon Shashua
• Shimon Ullman

Shimon Ullman We describe in this paper a network that performs grouping of image con(cid: 173) tours. The input to the net are fragments of image contours, and the output is the partitioning of the fragments into groups, together with a saliency measure for each group. The grouping is based on a measure of overall length and curvature. The network decomposes the overall opti(cid: 173) mization problem into independent optimal pairing problems performed at each node. The resulting computation maps into a uniform locally connected network of simple computing elements. 1 The Problenl: Contour Grouping A problem that often arises in visual information processing is the linking of con(cid: 173) tour fragments into optimal groups. For example, certain subsets of contours spon(cid: 173) taneously form perceptual groups, as illustrated in Fig. 1, and are often detected immediately without scanning the image in a systematic manner. Grouping process of this type are likely to play an important role in object recognition by segmenting the image and selecting image structures that are likely to correspond to objects of interest in the scene. 'Ve propose that some form of autonomous grouping is performed at an early stage based on geometrical characteristics, that are independent of the identity of objects to be selected. The grouping process is governed by the notion of saliency in a way that priority is given to forming salient groups at the expense of potentially less salient ones. This general notion can again be illustrated by Fig. 1; it appears that certain groups spontaneously emerge, while grouping decisions concerning the less salient parts of the image may remain unresolved. As we shall see, the computation below exhibits a similar behavior. We define a grouping of the image contours as the formation of a set of disjoint