FairyTED: A Fair Rating Predictor for TED Talk Data

Abstract

With the recent trend of applying machine learning in every aspect of human life, it is important to incorporate fairness into the core of the predictive algorithms. We address the problem of predicting the quality of public speeches while being fair with respect to sensitive attributes of the speakers, e.g. gender and race. We use the TED talks as an input repository of public speeches because it consists of speakers from a diverse community and has a wide outreach. Utilizing the theories of Causal Models, Counterfactual Fairness and state-of-the-art neural language models, we propose a mathematical framework for fair prediction of the public speaking quality. We employ grounded assumptions to construct a causal model capturing how different attributes affect public speaking quality. This causal model contributes in generating counterfactual data to train a fair predictive model. Our framework is general enough to utilize any assumption within the causal model. Experimental results show that while prediction accuracy is comparable to recent work on this dataset, our predictions are counterfactually fair with respect to a novel metric when compared to true data labels. The FairyTED setup not only allows organizers to make informed and diverse selection of speakers from the unobserved counterfactual possibilities but it also ensures that viewers and new users are not influenced by unfair and unbalanced ratings from arbitrary visitors to the ted.com website when deciding to view a talk.

Top-N Recommendation with Counterfactual User Preference Simulation

“…Suppose the users' feedback on an item pair (𝑖, 𝑗) is probabilistic, and we can observe 𝑖 > 𝑗 and 𝑖 < 𝑗 with the probabilities of 𝜂 and 1 −𝜂, respectively. Then, 𝜂 can actually measure the hardness of the sample, when 𝜂 is closer to 1 2 , then the sample is harder, since the propensity between the items is more ambiguous. Suppose 𝜖, 𝛿 ∈ (0, 1), and we use a simple voting mechanism to determine the relation between 𝑖 and 𝑗, then we need to have This theory suggests that we need to generate more samples (i.e., larger log 1 𝛿 2(1−2𝜂) 2 ) for the harder (i.e., smaller 𝜂) item pairs.…”

“…Remark. (1) In order to consider the potential noisy information and randomness in the recommender system, the structure equation models (i.e., 𝑭 ) are defined in a stochastic manner, which helps to learn user preference in a more accurate and robust manner. (2) The exogenous variables in the recommendation problem can be explained as the conditions (e.g., system status, user habit, etc.)…”

“…To begin with, we reformulate the recommendation problem by a structure causal model 𝑴 = {𝑮, 𝑭 }. In specific, the causal graph 𝑮 is defined as follows (see Figure 1(b)): (1) U, R and S are the nodes representing the user, the recommendation list and the positive items 1 selected by the user. (2) U → R encodes the fact that the recommendation list is generated according to the user preference.…”

“…In another research line, causal inference (CI) has been recently introduced into the machine learning community to augment the training data for more comprehensive model optimization [1]. The basic idea is firstly assuming an underlying structure causal model (SCM), and then learning the model parameters based on the observed data.…”

Modeling Language Usage and Listener Engagement in Podcasts

“…To understand how the relationship between linguistic features and engagement in podcasts compares to other spoken media, we carry out the same analysis on a corpus of 2480 talks from the TED Conferences (Tanveer et al, 2018;Acharyya et al, 2020). While we don't have access to the stream rate of the lectures, the data includes the total view count and ratings.…”