Multidimensional Gains for Stochastic Approximation

Saab, Samer S.; Shen, Dong

doi:10.1109/tnnls.2019.2920930

Cited by 16 publications

(3 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To determine the optimal set of parameters that fits the training data, the training model has to optimize a loss (objective) function, which penalizes the model when it produces an inaccurate label on a data point. Let l W; x i y i be the loss function for each data sample x i , with W being a matrix (or several matrices) of weights between neurons [18]. Correspondingly, the loss function on a client's local dataset D c is given in Equation (1).…”

Section: Federated Learningmentioning

confidence: 99%

Federated Machine Learning: Survey, Multi-Level Classification, Desirable Criteria and Future Directions in Communication and Networking Systems

Wahab

Mourad

Otrok

et al. 2021

IEEE Commun. Surv. Tutorials

375

110

View full text Add to dashboard Cite

The communication and networking field is hungry for machine learning decision-making solutions to replace the traditional model-driven approaches that proved to be not rich enough for seizing the ever-growing complexity and heterogeneity of the modern systems in the field. Traditional machine learning solutions assume the existence of (cloud-based) central entities that are in charge of processing the data. Nonetheless, the difficulty of accessing private data, together with the high cost of transmitting raw data to the central entity gave rise to a decentralized machine learning approach called Federated Learning. The main idea of federated learning is to perform an on-device collaborative training of a single machine learning model without having to share the raw training data with any third-party entity. Although few survey articles on federated learning already exist in the literature, the motivation of this survey stems from three essential observations. The first one is the lack of a fine-grained multi-level classification of the federated learning literature, where the existing surveys base their classification on only one criterion or aspect. The second observation is that the existing surveys focus only on some common challenges, but disregard other essential aspects such as reliable client selection, resource management and training service pricing. The third observation is the lack of explicit and straightforward directives for researchers to help them design future federated learning solutions that overcome the state-of-the-art research gaps. To address these points, we first provide a comprehensive tutorial on federated learning and its associated concepts, technologies and learning approaches. We then survey and highlight the applications and future directions of federated learning in the domain of communication and networking. Thereafter, we design a three-level classification scheme that first categorizes the federated learning literature based on the high-level challenge that they tackle. Then, we classify each high-level challenge into a set of specific low-level challenges to foster a better understanding of the topic. Finally, we provide, within each low-level challenge, a fine-grained classification based on the technique used to address this particular challenge. For each category of high-level challenges, we provide a set of desirable criteria and future research directions that are aimed to help the research community design innovative and efficient future solutions. To the best of our knowledge, our survey is the most comprehensive in terms of challenges and techniques it covers and the most fine-grained in terms of the multi-level classification scheme it presents.

show abstract

Section: Federated Learningmentioning

confidence: 99%

Federated Machine Learning: Survey, Multi-Level Classification, Desirable Criteria and Future Directions in Communication and Networking Systems

Wahab

Mourad

Otrok

et al. 2021

IEEE Commun. Surv. Tutorials

375

110

View full text Add to dashboard Cite

show abstract

“…Ref. [6] also assumes direct information on the Hessian is available in a second-order stochastic method, but allows for loss functions more general than the ERF. Ref.…”

Section: Introduction 1problem Contextmentioning

confidence: 99%

“…A floating-point operation is assumed to be either a summation or a multiplication, while transposition requires no FLOPs. For the updating H k step in original 2SPSA, 3p 2 FLOPs are required per (5) and 4p 2 FLOPs are required per (6). In the proposed implementation, 10p FLOPs are required to get ũk and ṽk per (11) and (12), respectively, and 22p 2 /6 + O(p) FLOPs are required to update the symmetric indefinite factorization of H k [24, Thm.…”

mentioning

confidence: 99%

Efficient Implementation of Second-Order Stochastic Approximation Algorithms in High-Dimensional Problems

Zhu

Wang

Spall³

2019

Preprint

View full text Add to dashboard Cite

Stochastic approximation (SA) algorithms have been widely applied in minimization problems when the loss functions and/or the gradient information are only accessible through noisy evaluations. Stochastic gradient (SG) descent-a first-order algorithm and a workhorse of much machine learning-is perhaps the most famous form of SA. Among all SA algorithms, the second-order simultaneous perturbation stochastic approximation (2SPSA) and the second-order stochastic gradient (2SG) are particularly efficient in handling high-dimensional problems, covering both gradient-free and gradient-based scenarios. However, due to the necessary matrix operations, the per-iteration floating-pointoperations (FLOPs) cost of the standard 2SPSA/2SG is O(p 3 ), where p is the dimension of the underlying parameter. Note that the O(p 3 ) FLOPs cost is distinct from the classical SPSA-based per-iteration O(1) cost in terms of the number of noisy function evaluations.In this work, we propose a technique to efficiently implement the 2SPSA/2SG algorithms via the symmetric indefinite matrix factorization and show that the FLOPs cost is reduced from O(p 3 ) to O(p 2 ). The formal almost sure convergence and rate of convergence for the newly proposed approach are directly inherited from the standard 2SPSA/2SG. The improvement in efficiency and numerical stability is demonstrated in two numerical studies.

show abstract

A novel approach for detection of APT malware using multi-dimensional hybrid Bayesian belief network

Sharma

Gupta

Singh

et al. 2022

Int. J. Inf. Secur.

View full text Add to dashboard Cite

Multidimensional Gains for Stochastic Approximation

Cited by 16 publications

References 47 publications

Federated Machine Learning: Survey, Multi-Level Classification, Desirable Criteria and Future Directions in Communication and Networking Systems

Federated Machine Learning: Survey, Multi-Level Classification, Desirable Criteria and Future Directions in Communication and Networking Systems

Efficient Implementation of Second-Order Stochastic Approximation Algorithms in High-Dimensional Problems

A novel approach for detection of APT malware using multi-dimensional hybrid Bayesian belief network

Contact Info

Product

Resources

About