Reward-Based 1-bit Compressed Federated Distillation on Blockchain

Abstract

The recent advent of various forms of Federated Knowledge Distillation (FD) paves the way for a new generation of robust and communication-efficient Federated Learning (FL), where mere soft-labels are aggregated, rather than whole gradients of Deep Neural Networks (DNN) as done in previous FL schemes. This security-perdesign approach in combination with increasingly performant Internet of Things (IoT) and mobile devices opens up a new realm of possibilities to utilize private data from industries as well as from individuals as input for artificial intelligence model training. Yet in previous FL systems, lack of trust due to the imbalance of power between workers and a central authority, the assumption of altruistic worker participation and the inability to correctly measure and compare contributions of workers hinder this technology from scaling beyond small groups of already entrusted entities towards mass adoption. This work aims to mitigate the aforementioned issues by introducing a novel decentralized federated learning framework where heavily compressed 1-bit soft-labels, resembling 1-hot label predictions, are aggregated on a smart contract. In a context where workers' contributions are now easily comparable, we modify the Peer Truth Serum for Crowdsourcing mechanism (PTSC) for FD to reward honest participation based on peer consistency in an incentive compatible fashion. Due to heavy reductions of both computational complexity and storage, our framework is a fully on-blockchain FL system that is feasible on simple smart contracts and therefore blockchain agnostic. We experimentally test our new framework and validate its theoretical properties.

Decentral and Incentivized Federated Learning Frameworks: A Systematic Literature Review

“…Customized and tailored blockchain solutions may be required with respect to the underlying use-case. Due to its restrictive scalability in terms of computation and storage, most of the analyzed FLFs apply blockchain as a complementary element in a more complex system, with a few exceptions [30], [39], [54], [58], [59]. Blockchain systems themselves are complex distributed systems, heterogeneous across many dimensions, yet can roughly be categorized into public, private and permissioned Blockchains.…”

“…Voting on the next leader (aggregator) [51], [52] or on each others contributions [58], [59], [68] further democratizes the process. Beyond explicit coordination operations like voting or leader selection, the implicit function of storing crucial information and data for the FL process, [30], [39], [56], [61] , verifying correctness of updates [34], [54] or keeping the registry of active members [30], [31], [38], [39], [58], [59] is crucial for the FL workflow and implies coordination through blockchain as an always accessible, verifiable, transparent and immutable infrastructure. 3) Payment Many general-purpose blockchain systems include cryptocurrency capabilities and therefore allow for the incorporation of instant, automatic and deterministic payment schemes defined by the smart contract's business logic.…”

“…Even though expansive, since all blockchain nodes store the same information in a redundant fashion, it might make sense to capitalize on the immutability and transparency feature of blockchain and store information where either a shared access among participants is required or where verifiability of the history is required to hold agents accountable for posterior reward calculations [36]. In particular, machine learning models [31], [37], [40], [44], [46], [47], [52], [53], [54], [58], [59], reputation scores [36], [42], User-information [30], [31], [38], [39], [58], [59] and Votes [30], [58], [59] are stored on-chain of the respective FLF.…”

“…To overcome the scalability-bottleneck of storage, some FLF applied an Interplanetary-File-System (IPFS) [71], where data is stored off-chain in a distributed file system, using the content-address as a unique pointer to each file in a global namespace over all computing devices [32], [34], [39], [42], [45], [50], [51], [60], [64]. Other FLF are based on novel design choices to tackle the scalability-issues: Witt et al [30] applied compressed Federated Knowledge Distillation, storing only 1-bit compressed soft-logits onchain. Chai et al [44] design a hierarchical FLF with two blockchain layers to reduce the computational overhead by outsourcing computation and storage to an application specific sub-chain.…”

“…For instance, Zhao et al propose a metric based on the Euclidean distance of workers' model updates [34]. Likewise, Witt et al propose that workers are to submit an unlabeled public dataset's labels predicted with their own local models and calculate the correlation of their labels as a metric [30]. This metric is integrated into the computation of reward distribution based on peer truth serum [74] for the FL setting.…”