Abdellatif Zaidi (Université Paris-Est, France); Shlomo Shamai (Shitz) (Technion, Israel)
Abstract:
This tutorial aims at providing a survey of recent progress in the area of distributed learning as well as its application in wireless communications. First we show that, perhaps in sharp contrast with the intuition, statistical learning can perform better when applied distributively, rather than in a centralized manner. In particular, it is shown that distributed learning has a smaller generalization error, i.e., is more robust to variations in the input data; and the improvement over centralized processing increases with the number of clients. We also discuss multi-round interactive learning models (such as with the popular Federated Learning) and show that more interactions with the parameter server does not result in a smaller generalization error. This suggest that local SGD in which learning is performed on small-sized batchs of data and then models are aggregated suitably is better than the popular SGD; and is in accordance with recent findings in this line of work which we will also discuss. In the second part of this tutorial we discuss architectures, training algorithms, network resources and applications. In particular, after recalling details of Federated Learning and Split Learning, and variants of them, we provide a comprehensive overview of the In-Network Learning, which can be seen as a generalization of the Information Bottleneck method to networks that can be modeled as directed Graphs. Finally, we discuss a number of connections with seemingly unrelated problems, such as communication over CRAN with oblivious relays. Also, we highlight a number of challenges and new directions.
Short bios:
Shlomo Shamai (Shitz) is with the Department of Electrical Engineering, Technion Israel Institute of Technology, where he is a Technion Distinguished Professor, and holds the William Fondiller Chair of Telecommunications. Dr. Shamai (Shitz) is an IEEE Life Fellow, an URSI Fellow, a member of the Israeli Academy of Sciences and Humanities and a foreign member of the US National Academy of Engineering. He is the recipient of the 2011 Claude E. Shannon Award, the 2014 Rothschild Prize in Mathematics/Computer Sciences and Engineering and the 2017 IEEE Richard W. Hamming Medal. He is a corecipient of the 2018 Third Bell Labs Prize for Shaping the Future of Information and Communications Technology. He is the recipient of numerous technical and paper awards and recognitions of the IEEE (Donald G. Fink Prize Paper Award), Information Theory, Communications and Signal Processing Societies as well as EURASIP.
Abdellatif Zaidi is with the department of Electrical Engineering, University of Gustave-Eiffel, France, where he is an associate professor. He is currently on leave at the Mathematics and Algorithmic Sciences Lab., France Research Center, Huawei France. Dr. Zaidi is recipient of the Research Excellence price in France ; and co-recipient (jointly with Shlomo Shamai (Shitz)) of the N# Best Paper Award in 2014. Dr. Zaidi is IEEE senior member. He served as Associate Editor for the journal IEEE Transactions on Wireless Communications and for the Eurasip Journal on Wireless Communications and Networking. He held several research visitor positions, including at the University of Notre Dame, Indiana, the Technical University of Munich and the Ecole Polytechnique Federale de Lausanne, EPFL. His research interests lie broadly in network information theory and its interactions with statistics, machine learning and wireless communications