Posts

I am happy to announce that our recent that studies Safe-EF algorithm for distributed non-smooth optimization with constraints has been accepted to ICML 2025! Quick summary of our contributions:
⚙️ Safe-EF algorithm which provably converges in the federated non-smooth optimization with constraints;
🚨 Algorithms like Compressed SGD and EF21 fail to converge in such a setting;
🛠️ Lower bounds for first-order methods in the compressed non-smooth convex setting.
🔄 High probability analysis in the stochastic setting;
🤖 Extensive experiments, including distributed humanoid training, show that Safe-EF efficiently minimizes the objective while preserving constraint violations. This is a joint work with Yarden As and Ilyas Fatkhullin. Check out our paper for more details!

1️⃣ The paper Unbiased and Sign Compression in Distributed Learning: Comparing Noise Resilience via SDEs (will be available online soon), joint work with Enea Monzio Compagnoni, Frank Norbert Proske, and Aurelien Lucchi got accepted to AISTATS 2025 as Oral!
2️⃣ Two papers Towards Faster Decentralized Stochastic Optimization with Communication Compression, joint work with Yuan Gao and Sebastian Stich, and Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise, joint work with Enea Monzio Compagnoni, Tianlin Liu, Frank Norbert Proske, Antonio Orvieto, Aurelien Lucchi, got accepted to ICLR.
3️⃣ And finally, the paper Partiall Personalized Federated Learning: Breaking the Curse of Data Heterogeneity, joint work with Konstantin Mishchenko, Eduard Gorbunov, Samuel Horváth, got accepted to TMLR.

I presented our recent work on loss landscape characterization during NeurIPS@Paris workhsop poster session. My travel was supported by the organizers of the workshop. In this work, we propose a class of functions that better captures the properties of loss landscape of neural networks.

I am happy to announce that I have one paper accepted to NeurIPS 2024 on a novel class of functions that neural networks represent. This class contains functions with saddles and local minima in contrast to previous works. This is a joint work with Niccolò Ajroldi , Antonio Orvieto, and Aurelien Lucchi. It is now available on arXiv.

New paper on decentralized training with contractive compressed communication. We achieve near-optimal convergence guarantees using Error Feedback and momentum tracking.

I attended KAUST Rising Stars in AI Symposium organized by KAUST AI Initiatve. As a part of the symposium, I gave a talk on AsGrad project.

I am thrilled to announce that AsGrad paper has been accepted to AISTATS 2024. In this paper we propose a unified framework to analyze asynchronous SGD-type algorithms covering many existing versions.

I am happy to announce that Econtrol paper has been accepted to ICLR 2024. In this paper we develop new Error Feedback mechanism to properly handle the bias coming from the compression and noise of stochastic gradients.

New paper on Error Feedback is now on arXiv. We developed new method, EControl, which provably handles two sources of noise – stochastic gradients and error from contractive compression – in all standard regimes. Moreover, in all settings the convergence improves with the number of clients participating in the training.

After ICLR deadline has passed, it is a right time to take a deep breath and post some updates. I am happy to share that I have finished my internship at Institute of Science and Technology Austria. I had a chance to work under the supervision of two great researchers, Mher Safaryan and Dan Alistarh. I presented the internship results during my Master’s thesis defense at Institut Polytechnique de Paris. Besides, I am also thrilled to announce that I am joining University of Basel as a new PhD student where I will work under the supervision of Aurelien Lucchi. I am looking forward to meet all members of the lab and start working on new challenging problems!

I am happy to announce that new paper has just been published on arXiv. It is a project in collaboration with amazing guys. The paper introduces over personalization phenomenon which allows to remove completely the effect of data heterogeneity and always takes place.

Right now I am doing my internship at Institute of Science and Technology Austria. Here I am working at Distributed Algorithms and Systems group, EPFL run by Prof. Dan Alistarh. I am working on theory of distributed SGD.

I am happy to announce that our recent paper was accepted to NeurIPS workshop HOO 2022: Order up! The Benefits of Higher-Order Optimization in Machine Learning. Besides, new paper on adaptive compression for Federated Learning has been just published on arXiv.

I am starting my second year of Master at IP Paris. This year my Master program is based on Data Science program of Ecole Polytechnique.

In our new paper we prove that a class of so called Three Point Compressors can be applied on Newton method. This result opens up a wide variety of communication strategies, such as a combination of contractive compression and lazy aggregation, available to our disposal to compress prohibitively costly curvature information.

I am very proud to announce that our FedNL paper was accepted to 39th International Conference on Machine Learning (ICML 2022).

Currenly I am in Lausanne. Here I have started my internship at Machine Learning and Optimization Lab, EPFL run by Prof. Martin Jaggi. I will work under the supervision of postdoctoral fellow Hadrien Hendrikx and Prof. Martin Jaggi.

Our paper: Basis Matters: Better Communication-Efficient Second Order Methods for Federated Learning, joint work with Xun Qian, Mher Safaryan, and Peter Richtàrik, was accepted to AISTATS 2022.

We present new techniques (Basis Learning) how to make Newton-type methods applicable to Federated Learning. The idea is based on choosing appropriate basis for local Hessians representations which can lead to much better compression mechanism. Particularly, if we choose standard basis in the space of square matrices, we cover FedNL method.

I have finally arrived to Paris. Here I am going to study at Institut Polytechnique de Paris supervised by Olivier Fercoq. My first year will be based on Applied Mathematics and Statistics Program.

Our FedNL paper has been accepted to ICML workshops: International Workshop on Federated Learning for User Privacy and Data Confidentiality and Beyond first-order methods in ML systems. The paper will be presented as a poster at the first workshop and as a prerecorded video at the second one.

I am attending PRAIRE/MIAI Artificial Intelligence Summer School (PAISS 2021) (this year virtually only). This summer school is organized by PRAIRIE, MIAI, NAVER LABS Europe, and INRIA. The summer school comprises lectures conducted by renowned experts in different areas of artificial intelligence. As a part of the summer school I have presented the poster on our FedNL paper.

I have given a talk based on our paper Distributed Second Order Methods with Fast Rates and Compressed Communication at Maths&AI: MIPT-UGA young researchers workshop. The recorded video is available on YouTube. Here you can find the slides that I prepared for the talk.

I am very happy to announce that I defended my Bachelor thesis and finished the undergraduate study at MIPT yesterday. Now I am planning to conitnue my education at Institut Polytechnique de Paris.

We propose a family of Federated Newton Learn (FedNL) methods, which we believe is a marked step in the direction of making second-order methods applicable to FL. We perform a variety of numerical experiments that show that our FedNL methods have state-of-the-art communication complexity when compared to key baselines.

Our paper Distributed Second Order Methods with Fast Rates and Compressed Communication has been accepted to ICML 2021, which will be run virtually during July 18-24, 2021.

I have presented our recent work Distributed Second Order Methods with Fast Rates and Compressed Communication at KAUST Conference on Artificial Intelligence. The recorded video is available on YouTube.

Peter Richtárik has given a talk on our recent paper: Distributed Second Order Methods with Fast Rates and Compressed Communication, on All Russian Optimization Seminar. The recorded video will be available soon on YouTube channel of the seminar. In addition, I have presented the poster on the paper at Communication Efficient Distributed Optimization Workshop.

I have gotten one more internship at Machine Learning and Optimization Laboratory that Prof. Peter Richtárik runs. We will continue our work on second-order methods for distributed optimization.

We developed new communication efficient second order methods with fast rates. Our results are supported with experimental results on real datasets, and show several orders of magnitude improvement on baseline and state-of-the-art methods in terms of communication complexity. You can find more information about the paper by clicking on the link.