Finetuning Analytics Information Systems for a Better Understanding of Users: Evidence of Personification Bias on Multiple Digital Channels

Abstract

Although the effect of hyperparameters on algorithmic outputs is well known in machine learning, the effects of hyperparameters on information systems that produce user or customer segments are relatively unexplored. This research investigates the effect of varying the number of user segments on the personification of user engagement data in a real analytics information system, employing the concept of persona. We increment the number of personas from 5 to 15 for a total of 330 personas and 33 persona generations. We then examine the effect of changing the hyperparameter on the gender, age, nationality, and combined gender-age-nationality representation of the user population. The results show that despite using the same data and algorithm, varying the number of personas strongly biases the information system’s personification of the user population. The hyperparameter selection for the 990 total personas results in an average deviation of 54.5% for gender, 42.9% for age, 28.9% for nationality, and 40.5% for gender-age-nationality. A repeated analysis of two other organizations shows similar results for all attributes. The deviation occurred for all organizations on all platforms for all attributes, as high as 90.9% in some cases. The results imply that decision makers using analytics information systems should be aware of the effect of hyperparameters on the set of user or customer segments they are exposed to. Organizations looking to effectively use persona analytics systems must be wary that altering the number of personas could substantially change the results, leading to drastically different interpretations about the actual user base.