Arrow Research search

Author name cluster

Rajesh Jayaram

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

23 papers
2 author rows

Possible papers

23

FOCS Conference 2025 Conference Paper

Approximating High-Dimensional Earth Mover's Distance as Fast as Closest Pair

  • Lorenzo Beretta 0001
  • Vincent Cohen-Addad
  • Rajesh Jayaram
  • Erik Waingarten

We give a reduction from (1 + ε)-approximate Earth Mover’s Distance (EMD) to (1 + ε)-approximate Closest Pair (CP). As a consequence, we improve the fastest known approximation algorithm for high-dimensional EMD. Here, given p ∈ [1], [2] and two sets of n points $X, Y \subset \left( {{\mathbb{R}^d}, {\ell _p}} \right)$, their EMD is the minimum cost of a perfect matching between X and Y, where the cost of matching two vectors is their ℓ p distance. Further, CP is the basic problem of finding a pair of points realizing min x∈X, y∈Y ║x − y║ p. Our contribution is twofold: • We show that if (1 + ε)-approximate CP can be computed in time n 2−ϕ, then a 1 + O(ε) approximation to EMD can be computed in time n 2−Ω(ϕ). • Plugging in the fastest known algorithm for CP [5], we obtain a (1 + ε)-approximation algorithm for EMD running in time ${n^{2 - \tilde \Omega \left( {{\varepsilon ^{1/3}}} \right)}}$ for high-dimensional point sets, which improves over the prior fastest running time of ${n^{2 - \Omega \left( {{\varepsilon ^2}} \right)}}$ [13]. Our main technical contribution is a sublinear implementation of the Multiplicative Weights Update framework for EMD. Specifically, we demonstrate that the updates can be executed without ever explicitly computing or storing the weights; instead, we exploit the underlying geometric structure to perform the updates implicitly.

NeurIPS Conference 2025 Conference Paper

Hierarchical Retrieval: The Geometry and a Pretrain-Finetune Recipe

  • Chong You
  • Rajesh Jayaram
  • Ananda Theertha Suresh
  • Robin Nittka
  • Felix Yu
  • Sanjiv Kumar

Dual encoder (DE) models, where a pair of matching query and document are embedded into similar vector representations, are widely used in information retrieval due to their simplicity and scalability. However, the Euclidean geometry of the embedding space limits the expressive power of DEs, which may compromise their quality. This paper investigates such limitations in the context of hierarchical retrieval (HR), where the document set has a hierarchical structure and the matching documents for a query are all of its ancestors. We first prove that DEs are feasible for HR as long as the embedding dimension is linear in the depth of the hierarchy and logarithmic in the number of documents. Then we study the problem of learning such embeddings in a standard retrieval setup where DEs are trained on samples of matching query and document pairs. Our experiments reveal a lost-in-the-long-distance phenomenon, where retrieval accuracy degrades for documents further away in the hierarchy. To address this, we introduce a pretrain-finetune recipe that significantly improves long-distance retrieval without sacrificing performance on closer documents. We experiment on a realistic hierarchy from WordNet for retrieving documents at various levels of abstraction, and show that pretrain-finetune boosts the recall on long-distance pairs from 19% to 76%. Finally, we demonstrate that our method improves retrieval of relevant products on a shopping queries dataset.

ICML Conference 2025 Conference Paper

Randomized Dimensionality Reduction for Euclidean Maximization and Diversity Measures

  • Jie Gao 0001
  • Rajesh Jayaram
  • Benedikt Kolbe
  • Shay Sapir
  • Chris Schwiegelshohn
  • Sandeep Silwal
  • Erik Waingarten

Randomized dimensionality reduction is a widely-used algorithmic technique for speeding up large-scale Euclidean optimization problems. In this paper, we study dimension reduction for a variety of maximization problems, including max-matching, max-spanning tree, as well as various measures for dataset diversity. For these problems, we show that the effect of dimension reduction is intimately tied to the doubling dimension $\lambda_X$ of the underlying dataset $X$—a quantity measuring intrinsic dimensionality of point sets. Specifically, the dimension required is $O(\lambda_X)$, which we also show is necessary for some of these problems. This is in contrast to classical dimension reduction results, whose dependence grow with the dataset size $|X|$. We also provide empirical results validating the quality of solutions found in the projected space, as well as speedups due to dimensionality reduction.

NeurIPS Conference 2024 Conference Paper

Efficient Centroid-Linkage Clustering

  • MohammadHossein Bateni
  • Laxman Dhulipala
  • Willem Fletcher
  • Kishen N. Gowda
  • D E. Hershkowitz
  • Rajesh Jayaram
  • Jakub Łącki

We give an algorithm for Centroid-Linkage Hierarchical Agglomerative Clustering (HAC), which computes a $c$-approximate clustering in roughly $n^{1+O(1/c^2)}$ time. We obtain our result by combining a new centroid-linkage HAC algorithm with a novel fully dynamic data structure for nearest neighbor search which works under adaptive updates. We also evaluate our algorithm empirically. By leveraging a state-of-the-art nearest-neighbor search library, we obtain a fast and accurate centroid-linkage HAC algorithm. Compared to an existing state-of-the-art exact baseline, our implementation maintains the clustering quality while delivering up to a $36\times$ speedup due to performing fewer distance comparisons.

ICLR Conference 2024 Conference Paper

HyperAttention: Long-context Attention in Near-Linear Time

  • Insu Han
  • Rajesh Jayaram
  • Amin Karbasi
  • Vahab Mirrokni
  • David P. Woodruff
  • Amir Zandieh

We present an approximate attention mechanism named `HyperAttention` to address the computational challenges posed by the growing complexity of long contexts used in Large Language Models (LLMs). Recent work suggests that in the worst-case scenario, the quadratic time is necessary unless the entries of the attention matrix are bounded or the matrix has low stable rank. We introduce two parameters which measure: (1) the max column norm in the normalized attention matrix, and (2) the ratio of row norms in the unnormalized attention matrix after detecting and removing large entries. We use these fine-grained parameters to capture the hardness of the problem. Despite previous lower bounds, we are able to achieve a linear time sampling algorithm even when the matrix has unbounded entries or a large stable rank, provided the above parameters are small. HyperAttention features a modular design that easily accommodates integration of other fast low-level implementations, particularly FlashAttention. Empirically, employing Locality Sensitive Hashing (LSH) to identify large entries, HyperAttention outperforms existing methods, giving significant speed improvements compared to state-of-the-art solutions like FlashAttention. This development presents substantial implications for enabling LLMs to handle significantly larger contexts.

SODA Conference 2024 Conference Paper

Massively Parallel Algorithms for High-Dimensional Euclidean Minimum Spanning Tree

  • Rajesh Jayaram
  • Vahab Mirrokni
  • Shyam Narayanan
  • Peilin Zhong

We study the classic Euclidean Minimum Spanning Tree (MST) problem in the Massively Parallel Computation (MPC) model. Given a set X ⊂ ℝ d of n points, the goal is to produce a spanning tree for X with weight within a small factor of optimal. Euclidean MST is one of the most fundamental hierarchical geometric clustering algorithms, and with the proliferation of enormous high-dimensional data sets, such as massive transformer-based embeddings, there is now a critical demand for efficient distributed algorithms to cluster such data sets. In low-dimensional space, where d = O (1), Andoni, Nikolov, Onak, and Yaroslavtsev [STOC ‘14] gave a constant round MPC algorithm that obtains a high accuracy (1 + ɛ)-approximate solution. However, the situation is much more challenging for high-dimensional spaces: the best-known algorithm to obtain a constant approximation requires O (log n ) rounds. Recently Chen, Jayaram, Levi, and Waingarten [STOC ‘22] gave a O (log n ) approximation algorithm in a constant number of rounds based on embeddings into tree metrics. However, to date, no known algorithm achieves both a constant number of rounds and approximation. In this paper, we make strong progress on this front by giving a constant factor approximation in Õ (log log n ) rounds of the MPC model. In contrast to tree-embedding-based approaches, which necessarily must pay Ω(log n )-distortion, our algorithm is based on a new combination of graph-based distributed MST algorithms and geometric space partitions. Additionally, although the approximate MST we return can have a large depth, we show that it can be modified to obtain a O (log log n )-round constant factor approximation to the Euclidean Traveling Salesman Problem (TSP) in the MPC model. Previously, only a O (log n ) round was known for the problem. * The full version of the paper can be accessed at https: //arxiv. org/abs/2308. 00503.

NeurIPS Conference 2024 Conference Paper

MUVERA: Multi-Vector Retrieval via Fixed Dimensional Encoding

  • Laxman Dhulipala
  • Majid Hadian
  • Rajesh Jayaram
  • Jason Lee
  • Vahab Mirrokni

Neural embedding models have become a fundamental component of modern information retrieval (IR) pipelines. These models produce a single embedding $x \in \mathbb{R}^d$ per data-point, allowing for fast retrieval via highly optimized maximum inner product search (MIPS) algorithms. Recently, beginning with the landmark ColBERT paper, multi-vector models, which produce a set of embedding per data point, have achieved markedly superior performance for IR tasks. Unfortunately, using these models for IR is computationally expensive due to the increased complexity of multi-vector retrieval and scoring. In this paper, we introduce MUVERA (MUlti-VEctor Retrieval Algorithm), a retrieval mechanism which reduces multi-vector similarity search to single-vector similarity search. This enables the usage of off-the-shelf MIPS solvers for multi-vector retrieval. MUVERA asymmetrically generates Fixed Dimensional Encodings (FDEs) of queries and documents, which are vectors whose inner product approximates multi-vector similarity. We prove that FDEs give high-quality $\epsilon$-approximations, thus providing the first single-vector proxy for multi-vector similarity with theoretical guarantees. Empirically, we find that FDEs achieve the same recall as prior state-of-the-art heuristics while retrieving 2-5$\times$ fewer candidates. Compared to prior state of the art implementations, MUVERA achieves consistently good end-to-end recall and latency across a diverse set of the BEIR retrieval datasets, achieving an average of 10% improved recall with 90% lower latency.

NeurIPS Conference 2023 Conference Paper

Near-Linear Time Algorithm for the Chamfer Distance

  • Ainesh Bakshi
  • Piotr Indyk
  • Rajesh Jayaram
  • Sandeep Silwal
  • Erik Waingarten

For any two point sets $A, B \subset \mathbb{R}^d$ of size up to $n$, the Chamfer distance from $A$ to $B$ is defined as $\texttt{CH}(A, B)=\sum_{a \in A} \min_{b \in B} d_X(a, b)$, where $d_X$ is the underlying distance measure (e. g. , the Euclidean or Manhattan distance). The Chamfer distance is a popular measure of dissimilarity between point clouds, used in many machine learning, computer vision, and graphics applications, and admits a straightforward $O(d n^2)$-time brute force algorithm. Further, Chamfer distance is often used as a proxy for the more computationally demanding Earth-Mover (Optimal Transport) Distance. However, the \emph{quadratic} dependence on $n$ in the running time makes the naive approach intractable for large datasets. We overcome this bottleneck and present the first $(1+\epsilon)$-approximate algorithm for estimating Chamfer distance with a near-linear running time. Specifically, our algorithm runs in time $O(nd \log (n)/\epsilon^2)$ and is implementable. Our experiments demonstrate that it is both accurate and fast on large high-dimensional datasets. We believe that our algorithm will open new avenues for analyzing large high-dimensional point clouds. We also give evidence that if the goal is to report a $(1+\epsilon)$-approximate mapping from $A$ to $B$ (as opposed to just its value), then any sub-quadratic time algorithm is unlikely to exist.

SODA Conference 2023 Conference Paper

Optimal Fully Dynamic k -Center Clustering for Adaptive and Oblivious Adversaries

  • MohammadHossein Bateni
  • Hossein Esfandiari
  • Hendrik Fichtenberger
  • Monika Henzinger
  • Rajesh Jayaram
  • Vahab Mirrokni
  • Andreas Wiese

In fully dynamic clustering problems, a clustering of a given data set in a metric space must be maintained while it is modified through insertions and deletions of individual points. In this paper, we resolve the complexity of fully dynamic k -center clustering against both adaptive and oblivious adversaries. Against oblivious adversaries, we present the first algorithm for fully dynamic k -center in an arbitrary metric space that maintains an optimal (2 + ε)-approximation in O ( k · polylog( n, Δ)) amortized update time. Here, n is an upper bound on the number of active points at any time, and Δ is the aspect ratio of the metric space. Previously, the best known amortized update time was O ( k 2 · polylog( n, Δ)), and is due to Chan, Gourqin, and Sozio (2018). Moreover, we demonstrate that our runtime is optimal up to polylog( n, Δ) factors. In fact, we prove that even offline algorithms for k -clustering tasks in arbitrary metric spaces, including k -medians, k -means, and k -center, must make at least Ω( nk ) distance queries to achieve any non-trivial approximation factor. This implies a lower bound of Ω( k ) which holds even for the insertions-only setting. For adaptive adversaries, we give the first deterministic algorithm for fully dynamic k -center which achieves a approximation in O ( k -polylog( n, Δ)) amortized update time. Further, we demonstrate that any algorithm which achieves a -approximation against adaptive adversaries requires f (k, n) update time, for any arbitrary function f. Thus, in the regime where, we close the complexity of the problem up to polylog( n, Δ) factors in the update time. Our lower bound extends to other k -clustering tasks in arbitrary metric spaces, including k -medians and k -means. Finally, despite the aforementioned lower bounds, we demonstrate that an update time sublinear in k is possible against oblivious adversaries for metric spaces which admit locally sensitive hash functions (LSH), resulting in improved algorithms for a large class of metrics including Euclidean space, ℓ p -spaces, the Hamming Metric, and the Jaccard Metric. We also give the first fully dynamic O (1)-approximation algorithms for the closely related k -sum-of-radii and k -sum-of-diameter problems, with O (poly( k, log Δ)) update time.

STOC Conference 2023 Conference Paper

Streaming Euclidean MST to a Constant Factor

  • Xi Chen 0001
  • Vincent Cohen-Addad
  • Rajesh Jayaram
  • Amit Levi
  • Erik Waingarten

We study streaming algorithms for the fundamental geometric problem of computing the cost of the Euclidean Minimum Spanning Tree (MST) on an n -point set X ⊂ ℝ d . In the streaming model, the points in X can be added and removed arbitrarily, and the goal is to maintain an approximation in small space. In low dimensions, (1+є) approximations are possible in sublinear space [Frahling, Indyk, Sohler, SoCG ’05]. However, for high dimensional spaces the best known approximation for this problem was Õ(log n ), due to [Chen, Jayaram, Levi, Waingarten, STOC ’22], improving on the prior O (log 2 n ) bound due to [Indyk, STOC ’04] and [Andoni, Indyk, Krauthgamer, SODA ’08]. In this paper, we break the logarithmic barrier, and give the first constant factor sublinear space approximation to Euclidean MST. For any є≥ 1, our algorithm achieves an Õ(є −2 ) approximation in n O (є) space. We complement this by proving that any single pass algorithm which obtains a better than 1.10-approximation must use Ω(√ n ) space, demonstrating that (1+є) approximations are not possible in high-dimensions, and that our algorithm is tight up to a constant. Nevertheless, we demonstrate that (1+є) approximations are possible in sublinear space with O (1/є) passes over the stream. More generally, for any α ≥ 2, we give a α-pass streaming algorithm which achieves a (1+ O (logα + 1/ α є)) approximation in n O (є) d O (1) space. All our streaming algorithms are linear sketches, and therefore extend to the massively-parallel computation model (MPC). Thus, our results imply the first (1+є)-approximation to Euclidean MST in a constant number of rounds in the MPC model. Previously, such a result was only known for low-dimensional space [Andoni, Nikolov, Onak, Yaroslavtsev, STOC ’15], or required either O (log n ) rounds or a O (log n ) approximation.

STOC Conference 2022 Conference Paper

New streaming algorithms for high dimensional EMD and MST

  • Xi Chen 0001
  • Rajesh Jayaram
  • Amit Levi
  • Erik Waingarten

We study streaming algorithms for two fundamental geometric problems: computing the cost of a Minimum Spanning Tree (MST) of an n -point set X ⊂ {1,2,…,Δ} d , and computing the Earth Mover Distance (EMD) between two multi-sets A , B ⊂ {1,2,…,Δ} d of size n . We consider the turnstile model, where points can be added and removed. We give a one-pass streaming algorithm for MST and a two-pass streaming algorithm for EMD, both achieving an approximation factor of Õ(log n ) and using ( n , d ,Δ)-space only. Furthermore, our algorithm for EMD can be compressed to a single pass with a small additive error. Previously, the best known sublinear-space streaming algorithms for either problem achieved an approximation of O (min{ log n , log(Δ d )} log n ). For MST, we also prove that any constant space streaming algorithm can only achieve an approximation of Ω(log n ), analogous to the Ω(log n ) lower bound for EMD.

NeurIPS Conference 2022 Conference Paper

Stars: Tera-Scale Graph Building for Clustering and Learning

  • CJ Carey
  • Jonathan Halcrow
  • Rajesh Jayaram
  • Vahab Mirrokni
  • Warren Schudy
  • Peilin Zhong

A fundamental procedure in the analysis of massive datasets is the construction of similarity graphs. Such graphs play a key role for many downstream tasks, including clustering, classification, graph learning, and nearest neighbor search. For these tasks, it is critical to build graphs which are sparse yet still representative of the underlying data. The benefits of sparsity are twofold: firstly, constructing dense graphs is infeasible in practice for large datasets, and secondly, the runtime of downstream tasks is directly influenced by the sparsity of the similarity graph. In this work, we present Stars: a highly scalable method for building extremely sparse graphs via two-hop spanners, which are graphs where similar points are connected by a path of length at most two. Stars can construct two-hop spanners with significantly fewer similarity comparisons, which are a major bottleneck for learning based models where comparisons are expensive to evaluate. Theoretically, we demonstrate that Stars builds a graph in nearly-linear time, where approximate nearest neighbors are contained within two-hop neighborhoods. In practice, we have deployed Stars for multiple data sets allowing for graph building at the Tera-Scale, i. e. , for graphs with hundreds of billions of nodes and tens of trillions of edges. We evaluate the performance of Stars for clustering and graph learning, and demonstrate 10~1000-fold improvements in pairwise similarity comparisons and significant running time speedups with negligible quality loss.

ICML Conference 2021 Conference Paper

In-Database Regression in Input Sparsity Time

  • Rajesh Jayaram
  • Alireza Samadian
  • David P. Woodruff
  • Peng Ye 0005

Sketching is a powerful dimensionality reduction technique for accelerating algorithms for data analysis. A crucial step in sketching methods is to compute a subspace embedding (SE) for a large matrix $A \in \mathbb{R}^{N \times d}$. SE’s are the primary tool for obtaining extremely efficient solutions for many linear-algebraic tasks, such as least squares regression and low rank approximation. Computing an SE often requires an explicit representation of $A$ and running time proportional to the size of $A$. However, if $A= T_1 \Join T_2 \Join …\Join T_m$ is the result of a database join query on several smaller tables $T_i \in \mathbb{R}^{n_i \times d_i}$, then this running time can be prohibitive, as $A$ itself can have as many as $O(n_1 n_2 \cdots n_m)$ rows. In this work, we design subspace embeddings for database joins which can be computed significantly faster than computing the join. For the case of a two table join $A = T_1 \Join T_2$ we give input-sparsity algorithms for computing subspace embeddings, with running time bounded by the number of non-zero entries in $T_1, T_2$. This results in input-sparsity time algorithms for high accuracy regression, significantly improving upon the running time of prior FAQ-based methods for regression. We extend our results to arbitrary joins for the ridge regression problem, also considerably improving the running time of prior methods. Empirically, we apply our method to real datasets and show that it is significantly faster than existing algorithms.

STOC Conference 2021 Conference Paper

When is approximate counting for conjunctive queries tractable?

  • Marcelo Arenas
  • Luis Alberto Croquevielle
  • Rajesh Jayaram
  • Cristian Riveros

Conjunctive queries are one of the most common class of queries used in database systems, and the best studied in the literature. A seminal result of Grohe, Schwentick, and Segoufin (STOC 2001) demonstrates that for every class G of graphs, the evaluation of all conjunctive queries whose underlying graph is in G is tractable if, and only if, G has bounded treewidth. In this work, we extend this characterization to the counting problem for conjunctive queries. Specifically, for every class C of conjunctive queries with bounded treewidth, we introduce the first fully polynomial-time randomized approximation scheme (FPRAS) for counting answers to a query in C, and the first polynomial-time algorithm for sampling answers uniformly from a query in C. As a corollary, it follows that for every class G of graphs, the counting problem for conjunctive queries whose underlying graph is in G admits an FPRAS if, and only if, G has bounded treewidth (unless BPP is different from P). In fact, our FPRAS is more general, and also applies to conjunctive queries with bounded hypertree width, as well as unions of such queries.

ICLR Conference 2020 Conference Paper

Span Recovery for Deep Neural Networks with Applications to Input Obfuscation

  • Rajesh Jayaram
  • David P. Woodruff
  • Qiuyi (Richard) Zhang

The tremendous success of deep neural networks has motivated the need to better understand the fundamental properties of these networks, but many of the theoretical results proposed have only been for shallow networks. In this paper, we study an important primitive for understanding the meaningful input space of a deep network: span recovery. For $k<n$, let $\mathbf{A} \in \mathbb{R}^{k \times n}$ be the innermost weight matrix of an arbitrary feed forward neural network $M: \mathbb{R}^n \to \mathbb{R}$, so $M(x)$ can be written as $M(x) = \sigma(\mathbf{A} x)$, for some network $\sigma: \mathbb{R}^k \to \mathbb{R}$. The goal is then to recover the row span of $\mathbf{A}$ given only oracle access to the value of $M(x)$. We show that if $M$ is a multi-layered network with ReLU activation functions, then partial recovery is possible: namely, we can provably recover $k/2$ linearly independent vectors in the row span of $\mathbf{A}$ using poly$(n)$ non-adaptive queries to $M(x)$. Furthermore, if $M$ has differentiable activation functions, we demonstrate that \textit{full} span recovery is possible even when the output is first passed through a sign or $0/1$ thresholding function; in this case our algorithm is adaptive. Empirically, we confirm that full span recovery is not always possible, but only for unrealistically thin layers. For reasonably wide networks, we obtain full span recovery on both random networks and networks trained on MNIST data. Furthermore, we demonstrate the utility of span recovery as an attack by inducing neural networks to misclassify data obfuscated by controlled random noise as sensical inputs.

FOCS Conference 2020 Conference Paper

Testing Positive Semi-Definiteness via Random Submatrices

  • Ainesh Bakshi
  • Nadiia Chepurko
  • Rajesh Jayaram

We study the problem of testing whether a matrix A ∈ \mathbbR n×n with bounded entries ( ||A||∞ ≤ 1) is positive semidefinite (PSD), or ε-far in Euclidean distance from the PSD cone, meaning that, where B\succeq 0 denotes that B is PSD. Our main algorithmic contribution is a non-adaptive tester which distinguishes between these cases using only ~O(1/ε 4 ) queries to the entries of A. 11 Throughout the paper, ~O(·) hides log(1/ε) factors. If instead of the Eucledian norm we considered the distance in spectral norm, we obtain the “ l∞-gap problem”, where A is either PSD or satisfies. For this related problem, we give a ~O(1/ε 2 ) query tester, which we show is optimal up to log(1/ε) factors. Both our testers randomly sample a collection of principal sub-matrices and check whether these sub-matrices are PSD. Consequentially, our algorithms achieve one-sided error: whenever they output that A is not PSD, they return a certificate that A has negative eigenvalues. We complement our upper bound for PSD testing with Eucledian norm distance by giving a ~Ω(1/ε 2 ) lower bound for any non-adaptive algorithm. Our lower bound construction is general, and can be used to derive lower bounds for a number of spectral testing problems. As an example of the applicability of our construction, we obtain a new ~Ω(1/ε 4 ) sampling lower bound for testing the Schatten-1 norm with a εn 1. 5 gap, extending a result of Balcan, Li, Woodruff, and Zhang [11]. In addition, our hard instance results in new sampling lower bounds for estimating the Ky-Fan Norm, and the cost of rank- k approximations, i. e. .

NeurIPS Conference 2019 Conference Paper

Optimal Sketching for Kronecker Product Regression and Low Rank Approximation

  • Huaian Diao
  • Rajesh Jayaram
  • Zhao Song
  • Wen Sun
  • David Woodruff

We study the Kronecker product regression problem, in which the design matrix is a Kronecker product of two or more matrices. Formally, given $A_i \in \R^{n_i \times d_i}$ for $i=1, 2, \dots, q$ where $n_i \gg d_i$ for each $i$, and $b \in \R^{n_1 n_2 \cdots n_q}$, let $\mathcal{A} = A_i \otimes A_2 \otimes \cdots \otimes A_q$. Then for $p \in [1, 2]$, the goal is to find $x \in \R^{d_1 \cdots d_q}$ that approximately minimizes $\|\mathcal{A}x - b\|_p$. Recently, Diao, Song, Sun, and Woodruff (AISTATS, 2018) gave an algorithm which is faster than forming the Kronecker product $\mathcal{A} \in \R^{n_1 \cdots n_q \times d_1 \cdots d_q}$. Specifically, for $p=2$ they achieve a running time of $O(\sum_{i=1}^q \texttt{nnz}(A_i) + \texttt{nnz}(b))$, where $ \texttt{nnz}(A_i)$ is the number of non-zero entries in $A_i$. Note that $\texttt{nnz}(b)$ can be as large as $\Theta(n_1 \cdots n_q)$. For $p=1, $ $q=2$ and $n_1 = n_2$, they achieve a worse bound of $O(n_1^{3/2} \text{poly}(d_1d_2) + \texttt{nnz}(b))$. In this work, we provide significantly faster algorithms. For $p=2$, our running time is $O(\sum_{i=1}^q \texttt{nnz}(A_i) )$, which has no dependence on $\texttt{nnz}(b)$. For $p<2$, our running time is $O(\sum_{i=1}^q \texttt{nnz}(A_i) + \texttt{nnz}(b))$, which matches the prior best running time for $p=2$. We also consider the related all-pairs regression problem, where given $A \in \R^{n \times d}, b \in \R^n$, we want to solve $\min_{x \in \R^d} \|\bar{A}x - \bar{b}\|_p$, where $\bar{A} \in \R^{n^2 \times d}, \bar{b} \in \R^{n^2}$ consist of all pairwise differences of the rows of $A, b$. We give an $O(\texttt{nnz}(A))$ time algorithm for $p \in[1, 2]$, improving the $\Omega(n^2)$ time required to form $\bar{A}$. Finally, we initiate the study of Kronecker product low rank and and low-trank approximation. For input $\mathcal{A}$ as above, we give $O(\sum_{i=1}^q \texttt{nnz}(A_i))$ time algorithms, which is much faster than computing $\mathcal{A}$.

FOCS Conference 2018 Conference Paper

Perfect Lp Sampling in a Data Stream

  • Rajesh Jayaram
  • David P. Woodruff

In this paper, we resolve the one-pass space complexity of L p sampling for p ∈ (0, 2). Given a stream of updates (insertions and deletions) to the coordinates of an underlying vector f ∈ ℝ n, a perfect Lp sampler must output an index i with probability |f i | p /||f|| p p, and is allowed to fail with some probability δ. So far, for p > 0 no algorithm has been shown to solve the problem exactly using poly(log n)-bits of space. In 2010, Monemizadeh and Woodruff introduced an approximate L p sampler, which outputs i with probability (1 ± ν)|f i | p /||f|| p p, using space polynomial in ν -1 and log(n). The space complexity was later reduced by Jowhari, Saglam, and Tardos to roughly O(ν -p log 2 n log δ -1 ) for p ∈ (0, 2), which tightly matches the Ω(log 2 n log δ -1 ) lower bound in terms of n and δ, but is loose in terms of ν. Given these nearly tight bounds, it is perhaps surprising that no lower bound at all exists in terms of ν-not even a bound of Ω(ν -1 ) is known. In this paper, we explain this phenomenon by demonstrating the existence of an O(log 2 n log δ -1 )-bit perfect L p sampler for p ∈ (0, 2). This shows that ν need not factor into the space of an L p sampler, which completely closes the complexity of the problem for this range of p. For p = 2, our bound is O(log 3 n log δ -1 )-bits, which matches the prior best known upper bound of O(ν -2 log 3 n log δ -1 ), but has no dependence on ν. Finally, we show that a (1 ± ε) relative error estimate of the frequency f i of the sampled index i can be obtained using an additional O(ε -p log n)-bits of space for p -2 log 2 n) bits for p = 2, which was possible before only by running the prior algorithms with ν = ε.

v2026.09.13