Scaling up psychology via Scientific Regret Minimization

Agrawal, Mayank; Peterson, Joshua C.; Griffiths, Thomas L.

doi:10.1073/pnas.1915841117

Cited by 73 publications

(52 citation statements)

References 73 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Bourgin et al (2019) took advantage of the larger sample sizes that can be obtained through virtual labs to scale up this approach, collecting human decisions for over 10,000 pairs of gambles. The resulting data set can be used to evaluate models of decision-making and is at a scale where machine learning methods can be used to augment the insights of human researchers (Agrawal, Peterson, and Griffiths 2020).…”

Section: Individualsmentioning

confidence: 99%

Scaling up experimental social, behavioral, and economic science

Almaatouq¹,

Becker²,

Bernstein³

et al. 2021

Preprint

View full text Add to dashboard Cite

The standard experimental paradigm in the social, behavioral, and economic sciences is extremely limited. Although recent advances in digital technologies and crowdsourcing services allow individual experiments to be deployed and run faster than in traditional physical labs, a majority of experiments still focus on one-off results that do not generalize easily to real-world contexts or even to other variations of the same experiment. As a result, there exist few universally acknowledged findings, and even those are occasionally overturned by new data. We argue that to achieve replicable, generalizable, scalable and ultimately useful social and behavioral science, a fundamental rethinking of the model of virtual-laboratory style experiments is required. Not only is it possible to design and run experiments that are radically different in scale and scope than was possible in an era of physical labs; this ability allows us to ask fundamentally different types of questions than have been asked historically of lab studies. We posit, however, that taking full advantage of this new and exciting potential will require four major changes to the infrastructure, methodology, and culture of experimental science: (1) significant investments in software design and participant recruitment, (2) innovations in experimental design and analysis of experimental data, (3) adoption of new models of collaboration, and (4) a new understanding of the nature and role of theory in experimental social and behavioral science. We conclude that the path we outline, although ambitious, is well within the power of current technology and has the potential to facilitate a new class of scientific advances in social, behavioral and economic studies.This paper emerged from discussions at a workshop held by the Computational Social Science Lab at the University of Pennsylvania in January 2020. The work was supported by James and Jane Manzi Analytics Fund and the Alfred P. Sloan Foundation.

show abstract

Section: Individualsmentioning

confidence: 99%

Scaling up experimental social, behavioral, and economic science

Almaatouq¹,

Becker²,

Bernstein³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The Hofman team provides two examples, one from Athey et al ( 2011) who come up with an explanatory model of bidding behavior in an auction and use it to predict outcomes that are then tested against the actual outcomes. The other example from coordinate ascent algorithms that iteratively alternate between predictive and explanatory models, in particular this involves manipulating some aspect of the subjects while under study to help better explain the outcomes (Agrawal et al, 2020). Somehow, such models should provide benefits that are greater than explanatory or predictive models done in isolation because they can predict "magnitude and direction of individual outcomes under changes or interventions" (Hofman et al 2021:Table 2).…”

Section: Integrative Modelingmentioning

confidence: 99%

Integrating Computer Prediction Methods in Social Science: A Comment on Hofman et al. (2021)

Breznau¹

2021

Preprint

View full text Add to dashboard Cite

Machine learning and other computer-driven prediction models are one of the fastest growing trends in computational social science. These methods and approaches were developed in computer science and with different goals and epistemologies than those in social science. The most obvious difference being a focus on prediction versus explanation. Predictive modeling offers great potential for improving research and theory development, but its adoption poses some challenges and creates new problems. For this reason, Hofman et al. (2021) published recommendations for more effective integration of predictive modeling into social science. In this communication I review their recommendations and expand on some additional concerns related to current practices and whether prediction can effectively serve the goals of most social scientists. Overall, I argue they provide a sound set of guidelines and a classification scheme that will serve those of us working in computational social science.

show abstract

“…They focused on risky choices and extensively studied issues in decision theory [39,40]. In addition, Agrawal et al proposed methodologies for building models and identifying novel phenomena in large datasets [41]. To overcome noise artifacts included in the datasets, they utilized sufficiently large datasets with data-driven models.…”

Section: Introductionmentioning

confidence: 99%

Deep Learning Approach to Generate a Synthetic Cognitive Psychology Behavioral Dataset

Choi¹,

Nah

et al. 2021

IEEE Access

View full text Add to dashboard Cite

Synthetic data generation is critical in machine and deep learning research to overcome the shortage of samples or dataset sizes. Various algorithms, including the generative adversarial network and autoencoder models, have been applied to generate artificial datasets in previous studies. In this study, we propose a synthetic data generation framework for a tabular dataset collected from cognitive psychology behavioral experiments based on deep learning algorithms. Tabular datasets for the Stroop task were used to develop our framework. On account of the relatively small sample size (N=102) of the dataset used in our study, we used a pre-trained generative adversarial network model to complement the size of the dataset. Furthermore, we proposed and applied five evaluation methods with statistical tests (overlapped sample test, constraint reflection test, correlation reflection test, distribution distance test, and feature distance test) to validate generation performance based on internal levels of table structure (instance level, feature level, and whole-set level evaluations). The proposed framework with a fine-tuned generative adversarial network algorithm was compared with a random generation method to verify generation performance, including the representation of the statistical characteristics of the original datasets. We found that the generated datasets from the proposed framework exhibited more similar statistical characteristics with the original dataset than the randomly generated datasets based on five evaluation methods. The results of this study provide not only generation algorithms for cognitive psychological datasets with tabular type but also a solution to the sample size issue for researchers.

show abstract

Scaling up psychology via Scientific Regret Minimization

Cited by 73 publications

References 73 publications

Scaling up experimental social, behavioral, and economic science

Scaling up experimental social, behavioral, and economic science

Integrating Computer Prediction Methods in Social Science: A Comment on Hofman et al. (2021)

Deep Learning Approach to Generate a Synthetic Cognitive Psychology Behavioral Dataset

Contact Info

Product

Resources

About