Ali Ghodsi

NeurIPS Conference 2024 Conference Paper

Orchid: Flexible and Data-Dependent Convolution for Sequence Modeling

• Mahdi Karami
• Ali Ghodsi

In the rapidly evolving field of deep learning, the demand for models that are both expressive and computationally efficient has never been more critical. This paper introduces Orchid, a novel architecture designed to address the quadratic complexity of traditional attention mechanisms without compromising the ability to capture long-range dependencies and in-context learning. At the core of this architecture lies a new data-dependent global convolution layer, which contextually adapts its kernel conditioned on input sequence using a dedicated conditioning neural network. We design two simple conditioning networks that maintain shift equivariance in our data-dependent convolution operation. The dynamic nature of the proposed convolution kernel grants Orchid high expressivity while maintaining quasilinear scalability for long sequences. We evaluate the proposed model across multiple domains, including language modeling and image classification, to highlight its performance and generality. Our experiments demonstrate that this architecture not only outperforms traditional attention-based architectures such as BERT and Vision Transformers with smaller model sizes, but also extends the feasible sequence length beyond the limitations of the dense attention layers. This achievement represents a significant step towards more efficient and scalable deep learning models for sequence modeling.

EAAI Journal 2017 Journal Article

Sparse supervised principal component analysis (SSPCA) for dimension reduction and variable selection

• Sara Sharifzadeh
• Ali Ghodsi
• Line H. Clemmensen
• Bjarne K. Ersbøll

Principal component analysis (PCA) 1 1 PCA: principal component analysis, SPCA: sparse PCA, SSPCA: sparse supervised PCA, SPLS: sparse partial least squares, PMD: penalized matrix decomposition, SVD: singular value decomposition, HSIC: Hilbert Schmidt independence criterion, RKHS: reproducing kernel Hilbert space, SIMPLS: statistically inspired modification of PLS, SVM: support vector machine, CV: cross validation, RBF: radial basis function, RMSE: root mean square error, ROI: region of interest, NIR: near infrared, SSC: solvable solid content, KNN: K nearest neighbour. is one of the main unsupervised pre-processing methods for dimension reduction. When the training labels are available, it is worth using a supervised PCA strategy. In cases that both dimension reduction and variable selection are required, sparse PCA (SPCA) methods are preferred. In this paper, a sparse supervised PCA (SSPCA) method is proposed for pre-processing. This method is appropriate especially in problems where, a high dimensional input necessitates the use of a sparse method and a target label is also available to guide the variable selection strategy. Such a method is valuable in many Engineering and scientific problems, when the number of training samples is also limited. The Hilbert Schmidt Independence Criteria (HSIC) is used to form an objective based on minimization of a loss function and an L 1 norm is used for regularization of the Eigen vectors. While the proposed objective function allows a sparse low rank solution for both linear and non-linear relationships between the input and response matrices, other similar methods in this case are only based on a linear model. The objective is solved based on penalized matrix decomposition (PMD) algorithm. We compare the proposed method with PCA, PMD-based SPCA and supervised PCA. In addition, SSPCA is also compared with sparse partial least squares (SPLS), due to the similarity between the two objective functions. Experimental results from the simulated as well as real data sets show that, SSPCA provides an appropriate trade-off between accuracy and sparsity. Comparisons show that, in terms of sparsity, SSPCA performs the highest level of variable reduction and also, in terms of accuracy it is one of the most successful methods. Therefore, the Eigen vectors found by SSPCA can be used for feature selection in various high dimensional problems.

AAAI Conference 2010 Conference Paper

Conformal Mapping by Computationally Efficient Methods

• Stefan Pintilie
• Ali Ghodsi

Dimensionality reduction is the process by which a set of data points in a higher dimensional space are mapped to a lower dimension while maintaining certain properties of these points relative to each other. One important property is the preservation of the three angles formed by a triangle consisting of three neighboring points in the high dimensional space. If this property is maintained for those same points in the lower dimensional embedding then the result is a conformal map. However, many of the commonly used nonlinear dimensionality reduction techniques, such as Locally Linear Embedding (LLE) or Laplacian Eigenmaps (LEM), do not produce conformal maps. Post-processing techniques formulated as instances of semi-definite programming (SDP) problems can be applied to the output of either LLE or LEM to produce a conformal map. However, the effectiveness of this approach is limited by the computational complexity of SDP solvers. This paper will propose an alternative postprocessing algorithm that produces a conformal map but does not require a solution to a SDP problem and so is more computationally efficient thus allowing it to be applied to a wider selection of datasets. Using this alternative solution, the paper will also propose a new algorithm for 3D object classification. An interesting feature of the 3D classification algorithm is that it is invariant to the scale and the orientation of the surface.

IJCAI Conference 2007 Conference Paper

• Ali Ghodsi
• Dana Wilkinson
• Finnegan Southey

Dimensionality reduction is a much-studied task in machine learning in which high-dimensional data is mapped, possibly via a non-linear transformation, onto a low-dimensional manifold. The resulting embeddings, however, may fail to capture features of interest. One solution is to learn a distance metric which prefers embeddings that capture the salient features. We propose a novel approach to learning a metric from side information to guide the embedding process. Our approach admits the use of two kinds of side information. The first kind is class-equivalence information, where some limited number of pairwise "same/different class" statements are known. The second form of side information is a limited set of distances between pairs of points in the target metric space. We demonstrate the effectiveness of the method by producing embeddings that capture features of interest.

NeurIPS Conference 2002 Conference Paper

Regularized Greedy Importance Sampling

• Finnegan Southey
• Dale Schuurmans
• Ali Ghodsi

Greedy importance sampling is an unbiased estimation technique that re- duces the variance of standard importance sampling by explicitly search- ing for modes in the estimation objective. Previous work has demon- strated the feasibility of implementing this method and proved that the technique is unbiased in both discrete and continuous domains. In this paper we present a reformulation of greedy importance sampling that eliminates the free parameters from the original estimator, and introduces a new regularization strategy that further reduces variance without com- promising unbiasedness. The resulting estimator is shown to be effective for difficult estimation problems arising in Markov random field infer- ence. In particular, improvements are achieved over standard MCMC estimators when the distribution has multiple peaked modes.