New paper on Error Feedback is now on arXiv. We developed new method, EControl, which provably handles two sources of noise – stochastic gradients and error from contractive compression – in all standard regimes. Moreover, in all settings the convergence improves with the number of clients participating in the training.
After ICLR deadline has passed, it is a right time to take a deep breath and post some updates. I am happy to share that I have finished my internship at Institute of Science and Technology Austria. I had a chance to work under the supervision of two great researchers, Mher Safaryan and Dan Alistarh. I presented the internship results during my Master’s thesis defense at Institut Polytechnique de Paris. Besides, I am also thrilled to announce that I am joining University of Basel as a new PhD student where I will work under the supervision of Aurelien Lucchi. I am looking forward to meet all members of the lab and start working on new challenging problems!
I am happy to announce that new paper has just been published on arXiv. It is a project in collaboration with amazing guys. The paper introduces over personalization phenomenon which allows to remove completely the effect of data heterogeneity and always takes place.
I am happy to announce that our recent paper was accepted to NeurIPS workshop HOO 2022: Order up! The Benefits of Higher-Order Optimization in Machine Learning. Besides, new paper on adaptive compression for Federated Learning has been just published on arXiv.
I am starting my second year of Master at IP Paris. This year my Master program is based on Data Science program of Ecole Polytechnique.
In our new paper we prove that a class of so called Three Point Compressors can be applied on Newton method. This result opens up a wide variety of communication strategies, such as a combination of contractive compression and lazy aggregation, available to our disposal to compress prohibitively costly curvature information.
I am very proud to announce that our FedNL paper was accepted to 39th International Conference on Machine Learning (ICML 2022).
Our paper: Basis Matters: Better Communication-Efficient Second Order Methods for Federated Learning, joint work with Xun Qian, Mher Safaryan, and Peter Richtàrik, was accepted to AISTATS 2022.
We present new techniques (Basis Learning) how to make Newton-type methods applicable to Federated Learning. The idea is based on choosing appropriate basis for local Hessians representations which can lead to much better compression mechanism. Particularly, if we choose standard basis in the space of square matrices, we cover FedNL method.
Our FedNL paper has been accepted to ICML workshops: International Workshop on Federated Learning for User Privacy and Data Confidentiality and Beyond first-order methods in ML systems. The paper will be presented as a poster at the first workshop and as a prerecorded video at the second one.
I am attending PRAIRE/MIAI Artificial Intelligence Summer School (PAISS 2021) (this year virtually only). This summer school is organized by PRAIRIE, MIAI, NAVER LABS Europe, and INRIA. The summer school comprises lectures conducted by renowned experts in different areas of artificial intelligence. As a part of the summer school I have presented the poster on our FedNL paper.
I have given a talk based on our paper Distributed Second Order Methods with Fast Rates and Compressed Communication at Maths&AI: MIPT-UGA young researchers workshop. The recorded video is available on YouTube. Here you can find the slides that I prepared for the talk.
We propose a family of Federated Newton Learn (FedNL) methods, which we believe is a marked step in the direction of making second-order methods applicable to FL. We perform a variety of numerical experiments that show that our FedNL methods have state-of-the-art communication complexity when compared to key baselines.
Our paper Distributed Second Order Methods with Fast Rates and Compressed Communication has been accepted to ICML 2021, which will be run virtually during July 18-24, 2021.
I have presented our recent work Distributed Second Order Methods with Fast Rates and Compressed Communication at KAUST Conference on Artificial Intelligence. The recorded video is available on YouTube.
Peter Richtárik has given a talk on our recent paper: Distributed Second Order Methods with Fast Rates and Compressed Communication, on All Russian Optimization Seminar. The recorded video will be available soon on YouTube channel of the seminar. In addition, I have presented the poster on the paper at Communication Efficient Distributed Optimization Workshop.
I have gotten one more internship at Machine Learning and Optimization Laboratory that Prof. Peter Richtárik runs. We will continue our work on second-order methods for distributed optimization.
We developed new communication efficient second order methods with fast rates. Our results are supported with experimental results on real datasets, and show several orders of magnitude improvement on baseline and state-of-the-art methods in terms of communication complexity. You can find more information about the paper by clicking on the link.