Big-team science does not guarantee generalizability

Abstract

A new era of global ‘big-team science’studies has transformed human behaviour research. These innovative studies rely on a large, distributed network of participants from different parts of the world and represent a substantial advancement over the average study in psychology that rarely goes beyond a single demographic population (for example, North American undergraduates) 1. Here we examine one such big-team science project that claimed the ‘globalizability’of temporal discounting, the phenomenon in which the subjective value of deferred rewards is smaller than that of immediate rewards 2. We argue that, although this study represents a substantial advance over the typical psychology study in its sampling approach, claims of global generalizability are overstated given the samples collected. Although the project recruited 171 researchers from 109 institutions, and 13,629 research participants speaking 40 languages across 61 countries, relying solely on the typical big-team methodology created an illusion of generalizability, leading authors to overestimate the extent to which research findings can be applied globally. Across the low-and-middle-income countries (LMICs) and high-income countries (HICs) included in Ruggeri et al. 2, we found that the samples were all similarly young, well educated, urban and digitally connected. This homogeneity belies the heterogeneity present within each country 3, 4. To avoid this illusion of generalizability, we argue that researchers should carefully consider three dimensions of diversity sample, author and methodological diversity.

Publication
Nature Human Behaviour
Click the Cite button above to demo the feature to enable visitors to import publication metadata into their reference management software.
Click the Journal Article button below abstract to check all other Journal Article in the website.