An emerging pattern shows older Internet users interact with and respond to social media very differently from younger users. As older Americans become the fastest-growing group of Internet and social media users, that heterogeneity will become central to online politics. Many influential online experiments, however, were run on samples with too few older adults to represent today’s Internet population — a problem that grows for studies conducted even a few years ago.
🔎 What Was Tested
- Concern: whether treatment effects from social media experiments vary by age or digital literacy, and whether common online samples miss that variation.
- Context: two Facebook experiments originally conducted on Amazon’s Mechanical Turk (MTurk), a sample known to underrepresent older users.
🧪 How the Replication Was Conducted
- Replicated the two original Facebook experiments using an online subject source that contains sufficient age variation to capture older users.
- Added a standard battery of questions to explicitly measure digital literacy for each subject.
📊 Key Findings
- Significant treatment-effect heterogeneity by both age and digital literacy was detected in the replication of one of the two experiments.
- The other experiment did not show significant heterogeneity by these measures.
- This pattern illustrates a limitation to generalizability when subject selection is related to treatment-effect heterogeneity: samples like MTurk can miss or distort effects that vary with age or digital skills.
📌 Why This Matters
- Practical implication: Mechanical Turk is not an appropriate recruitment source when researchers have reason to suspect treatment-effect heterogeneity by age or digital literacy — a likely concern for studies of digital media effects.
- Broader implication: researchers studying online political communication should prioritize samples with broader age and digital-literacy variation to ensure findings generalize to the current population of Internet users.
