Fighting Fake News in Encrypted Messaging with the Fuzzy Anonymous Complaint Tally System (FACTS)

Abstract

Recent years have seen a strong uptick in both the prevalence and real-world consequences of false information spread through online platforms. At the same time, encrypted messaging systems such as WhatsApp, Signal, and Telegram, are rapidly gaining popularity as users seek increased privacy in their digital lives. The challenge we address is how to combat the viral spread of misinformation without compromising privacy. Our FACTS system tracks user complaints on messages obliviously, only revealing the message's contents and originator once sufficiently many complaints have been lodged. Our system is private, meaning it does not reveal anything about the senders or contents of messages which have received few or no complaints; secure, meaning there is no way for a malicious user to evade the system or gain an outsized impact over the complaint system; and scalable, as we demonstrate excellent practical efficiency for up to millions of complaints per day. Our main technical contribution is a new collaborative counting Bloom filter, a simple construction with difficult probabilistic analysis, which may have independent interest as a privacy-preserving randomized count sketch data structure. Compared to prior work on message flagging and tracing in end-to-end encrypted messaging, our novel contribution is the addition of a high threshold of multiple complaints that are needed before a message is audited or flagged. We present and carefully analyze the probabilistic performance of our data structure, provide a precise security definition and proof, and then measure the accuracy and scalability of our scheme via experimentation.

SoK: Content Moderation for End-to-End Encryption

“…In these designs, code inspection and continued use should demonstrate the correct functionality of the system; there is no remote server that needs periodic inspection or auditing. A privacy improvement can also be achieved even for partial client privacy, by reducing the amount of information that is sent to the server [185] or by performing detection only after a threshold of problematic content was detected [33,231].…”

“…Message franking [62,98,151,164,167,173,183,222,359,376] (total: 10) Reveal source, traceback, or popular messages [173,231,285,360] (total: 4) Other user reporting [26,86,128,192,207,214,237,245,248,377,384] (total: 11)…”

“…We did not see any other examples of such methods being applied, and we would expect them to be inappropriate for most problem contexts. For misinformation, two works specifically attempted to detect or limit "viral" disinformation [118,231] rather than general misinformation. We hope to see more research and innovation on this topic both in and out of the encrypted setting.…”

“…In a different take on user reporting, Liu et al [231] described an approach for revealing messages once they reached a specific threshold of reports globally across all users, by constructing a "collaborative counting bloom filter. " Their goal was to reveal misinformation that was especially "viral" and thus by definition reached a large number of users; after this the server would be free to take action on those specific images (e.g.…”