Author name cluster

Mikael Johansson

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

11 papers

1 author row

NeurIPS Conference 2024 Conference Paper

Nonconvex Federated Learning on Compact Smooth Submanifolds With Heterogeneous Data

Jiaojiao Zhang
Jiang Hu
Anthony Man-Cho So
Mikael Johansson

Many machine learning tasks, such as principal component analysis and low-rank matrix completion, give rise to manifold optimization problems. Although there is a large body of work studying the design and analysis of algorithms for manifold optimization in the centralized setting, there are currently very few works addressing the federated setting. In this paper, we consider nonconvex federated learningover a compact smooth submanifold in the setting of heterogeneous client data. We propose an algorithm that leverages stochastic Riemannian gradients and a manifold projection operator to improve computational efficiency, uses local updates to improve communication efficiency, and avoids client drift. Theoretically, we show that our proposed algorithm converges sub-linearly to a neighborhood of a first-order optimal solution by using a novel analysis that jointly exploits the manifold structure and properties of the loss functions. Numerical experiments demonstrate that our algorithm has significantly smaller computational and communication overhead than existing methods.

PDF Details DOI

TMLR Journal 2023 Journal Article

Adaptive Hyperparameter Selection for Differentially Private Gradient Descent

Dominik Fay
Sindri Magnússon
Jens Sjölund
Mikael Johansson

We present an adaptive mechanism for hyperparameter selection in differentially private optimization that addresses the inherent trade-off between utility and privacy. The mechanism eliminates the often unstructured and time-consuming manual effort of selecting hyperparameters and avoids the additional privacy costs that hyperparameter selection otherwise incurs on top of that of the actual algorithm. We instantiate our mechanism for noisy gradient descent on non-convex, convex and strongly convex loss functions, respectively, to derive schedules for the noise variance and step size. These schedules account for the properties of the loss function and adapt to convergence metrics such as the gradient norm. When using these schedules, we show that noisy gradient descent converges at essentially the same rate as its noise-free counterpart. Numerical experiments show that the schedules consistently perform well across a range of datasets without manual tuning.

PDF Details

JMLR Journal 2023 Journal Article

Asynchronous Iterations in Optimization: New Sequence Results and Sharper Algorithmic Guarantees

Hamid Reza Feyzmahdavian
Mikael Johansson

We introduce novel convergence results for asynchronous iterations that appear in the analysis of parallel and distributed optimization algorithms. The results are simple to apply and give explicit estimates for how the degree of asynchrony impacts the convergence rates of the iterates. Our results shorten, streamline and strengthen existing convergence proofs for several asynchronous optimization methods and allow us to establish convergence guarantees for popular algorithms that were thus far lacking a complete theoretical understanding. Specifically, we use our results to derive better iteration complexity bounds for proximal incremental aggregated gradient methods, to obtain tighter guarantees depending on the average rather than maximum delay for the asynchronous stochastic gradient descent method, to provide less conservative analyses of the speedup conditions for asynchronous block-coordinate implementations of Krasnoselskii–Mann iterations, and to quantify the convergence rates for totally asynchronous iterations under various assumptions on communication delays and update rates. [abs] [ pdf ][ bib ] &copy JMLR 2023. ( edit, beta )

PDF Details

NeurIPS Conference 2023 Conference Paper

Bringing regularized optimal transport to lightspeed: a splitting method adapted for GPUs

Jacob Lindbäck
Zesen Wang
Mikael Johansson

We present an efficient algorithm for regularized optimal transport. In contrast toprevious methods, we use the Douglas-Rachford splitting technique to developan efficient solver that can handle a broad class of regularizers. The algorithmhas strong global convergence guarantees, low per-iteration cost, and can exploitGPU parallelization, making it considerably faster than the state-of-the-art formany problems. We illustrate its competitiveness in several applications, includingdomain adaptation and learning of generative models.

PDF Details

YNIMG Journal 2023 Journal Article

Episodic memory formation in unrestricted viewing

Andrey R. Nikolaev
Inês Bramão
Roger Johansson
Mikael Johansson

The brain systems of episodic memory and oculomotor control are tightly linked, suggesting a crucial role of eye movements in memory. But little is known about the neural mechanisms of memory formation across eye movements in unrestricted viewing behavior. Here, we leverage simultaneous eye tracking and EEG recording to examine episodic memory formation in free viewing. Participants memorized multi-element events while their EEG and eye movements were concurrently recorded. Each event comprised elements from three categories (face, object, place), with two exemplars from each category, in different locations on the screen. A subsequent associative memory test assessed participants' memory for the between-category associations that specified each event. We used a deconvolution approach to overcome the problem of overlapping EEG responses to sequential saccades in free viewing. Brain activity was time-locked to the fixation onsets, and we examined EEG power in the theta and alpha frequency bands, the putative oscillatory correlates of episodic encoding mechanisms. Three modulations of fixation-related EEG predicted high subsequent memory performance: (1) theta increase at fixations after between-category gaze transitions, (2) theta and alpha increase at fixations after within-element gaze transitions, (3) alpha decrease at fixations after between-exemplar gaze transitions. Thus, event encoding with unrestricted viewing behavior was characterized by three neural mechanisms, manifested in fixation-locked theta and alpha EEG activity that rapidly turned on and off during the unfolding eye movement sequences. These three distinct neural mechanisms may be the essential building blocks that subserve the buildup of coherent episodic memories during unrestricted viewing behavior.

Details DOI

AAAI Conference 2021 Conference Paper

A Flexible Framework for Communication-Efficient Machine Learning

Sarit Khirirat
Sindri Magnússon
Arda Aytekin
Mikael Johansson

With the increasing scale of machine learning tasks, it has become essential to reduce the communication between computing nodes. Early work on gradient compression focused on the bottleneck between CPUs and GPUs, but communicationefficiency is now needed in a variety of different system architectures, from high-performance clusters to energyconstrained IoT devices. In the current practice, compression levels are typically chosen before training and settings that work well for one task may be vastly sub-optimal for another dataset on another architecture. In this paper, we propose a flexible framework which adapts the compression level to the true gradient at each iteration, maximizing the improvement in the objective function that is achieved per communicated bit. Our framework is easy to adapt from one technology to the next by modeling how the communication cost depends on the compression level for the specific technology. Theoretical results and practical experiments indicate that the automatic tuning strategies significantly increase communication efficiency on several state-of-the-art compression schemes.

PDF Details

NeurIPS Conference 2021 Conference Paper

On the Convergence of Step Decay Step-Size for Stochastic Optimization

Xiaoyu Wang
Sindri Magnússon
Mikael Johansson

The convergence of stochastic gradient descent is highly dependent on the step-size, especially on non-convex problems such as neural network training. Step decay step-size schedules (constant and then cut) are widely used in practice because of their excellent convergence and generalization qualities, but their theoretical properties are not yet well understood. We provide convergence results for step decay in the non-convex regime, ensuring that the gradient norm vanishes at an $\mathcal{O}(\ln T/\sqrt{T})$ rate. We also provide near-optimal (and sometimes provably tight) convergence guarantees for general, possibly non-smooth, convex and strongly convex problems. The practical efficiency of the step decay step-size is demonstrated in several large-scale deep neural network training tasks.

PDF Details

NeurIPS Conference 2018 Conference Paper

Continuous-time Value Function Approximation in Reproducing Kernel Hilbert Spaces

Motoya Ohnishi
Masahiro Yukawa
Mikael Johansson
Masashi Sugiyama

Motivated by the success of reinforcement learning (RL) for discrete-time tasks such as AlphaGo and Atari games, there has been a recent surge of interest in using RL for continuous-time control of physical systems (cf. many challenging tasks in OpenAI Gym and DeepMind Control Suite). Since discretization of time is susceptible to error, it is methodologically more desirable to handle the system dynamics directly in continuous time. However, very few techniques exist for continuous-time RL and they lack flexibility in value function approximation. In this paper, we propose a novel framework for model-based continuous-time value function approximation in reproducing kernel Hilbert spaces. The resulting framework is so flexible that it can accommodate any kind of kernel-based approach, such as Gaussian processes and kernel adaptive filters, and it allows us to handle uncertainties and nonstationarity without prior knowledge about the environment or what basis functions to employ. We demonstrate the validity of the presented framework through experiments.

PDF Details

NeurIPS Conference 2018 Conference Paper

The Convergence of Sparsified Gradient Methods

Dan Alistarh
Torsten Hoefler
Mikael Johansson
Nikola Konstantinov
Sarit Khirirat
Cedric Renggli

Distributed training of massive machine learning models, in particular deep neural networks, via Stochastic Gradient Descent (SGD) is becoming commonplace. Several families of communication-reduction methods, such as quantization, large-batch methods, and gradient sparsification, have been proposed. To date, gradient sparsification methods--where each node sorts gradients by magnitude, and only communicates a subset of the components, accumulating the rest locally--are known to yield some of the largest practical gains. Such methods can reduce the amount of communication per step by up to \emph{three orders of magnitude}, while preserving model accuracy. Yet, this family of methods currently has no theoretical justification. This is the question we address in this paper. We prove that, under analytic assumptions, sparsifying gradients by magnitude with local error correction provides convergence guarantees, for both convex and non-convex smooth objectives, for data-parallel SGD. The main insight is that sparsification methods implicitly maintain bounds on the maximum impact of stale updates, thanks to selection by magnitude. Our analysis and empirical validation also reveal that these methods do require analytical conditions to converge well, justifying existing heuristics.

PDF Details

YNIMG Journal 2012 Journal Article

Growth of language-related brain areas after foreign language learning

Johan Mårtensson
Johan Eriksson
Nils Christian Bodammer
Magnus Lindgren
Mikael Johansson
Lars Nyberg
Martin Lövdén

The influence of adult foreign-language acquisition on human brain organization is poorly understood. We studied cortical thickness and hippocampal volumes of conscript interpreters before and after three months of intense language studies. Results revealed increases in hippocampus volume and in cortical thickness of the left middle frontal gyrus, inferior frontal gyrus, and superior temporal gyrus for interpreters relative to controls. The right hippocampus and the left superior temporal gyrus were structurally more malleable in interpreters acquiring higher proficiency in the foreign language. Interpreters struggling relatively more to master the language displayed larger gray matter increases in the middle frontal gyrus. These findings confirm structural changes in brain regions known to serve language functions during foreign-language acquisition.

Details DOI

YNIMG Journal 2011 Journal Article

Electrophysiological correlates of retrieval orientation in reality monitoring

Timm Rosburg
Axel Mecklinger
Mikael Johansson

Retrieval orientation describes the modulation in the processing of retrieval cues by the nature of the targeted material in memory. Retrieval orientation is usually investigated by analyzing the cortical responses to new (unstudied) material when different memory contents are targeted. This approach avoids confounding effects of retrieval success. We investigated the neural correlates of retrieval orientation in reality monitoring with event-related potentials (ERPs) and assessed the impact of retrieval accuracy on obtained ERP measures. Thirty-two subjects studied visually presented object names that were followed either by a picture of that object (perceived condition) or by the instruction to mentally generate such a picture (imagine condition). Subsequently, subjects had to identify object names of one study condition and reject object names of the second study condition together with newly presented object names. The data analysis showed that object names were more accurately identified when they had been presented in the perceived condition. Two topographically distinct ERP effects of retrieval orientation were revealed: From 600 to 1100ms after stimulus representation, ERPs were more positive at frontal electrode sites when object names from the imagine condition were targeted. The analysis of response-locked ERP data revealed an additional effect at posterior electrode sites, with more negative ERPs shortly after response onset when items from the imagine condition were targeted. The ERP effect at frontal electrode sites, but not at posterior electrode sites was modulated by relative memory accuracy, with stronger effects in subjects who had lower memory accuracy for items of the imagine condition. The findings are suggestive for a contribution of frontal brain areas to retrieval orientation processes in reality monitoring and indicate that neural correlates of retrieval orientation can be modulated by retrieval effort, with stronger activation of these correlates with increasing task demands.

Details DOI