When Algorithms Read Politics: Promise and Pitfalls of Text-as-Data

📌 What This Review Covers

Politics and political conflict are often expressed in written and spoken words, but the massive costs of analyzing even moderately sized text collections have limited their use in political science. Automated text analysis substantially reduces those costs and has already delivered on part of that promise, making large-scale study of political texts more feasible.

🔎 What the Review Finds

• Automated methods can cut the costs of analyzing large bodies of political text and have begun to produce useful results.
• These methods are not a substitute for careful thought and close reading: they require extensive, problem-specific validation before their outputs can be trusted.
• Common problems in the literature include misconceptions about what automated tools deliver and errors in implementation or interpretation.

🧭 What This Guide Provides

• A survey of a wide range of new automated content-analysis methods relevant to political texts.
• Practical guidance on how to validate model outputs so that automated inferences are reliable for substantive political inquiry.
• Clarification of common misconceptions and errors that have appeared in prior work.

⚠️ Key Caution and Recommendation

For automated text methods to become standard tools in political science, methodologists must develop both new algorithms and new, rigorous methods of validation tailored to political questions. Without this methodological work, automated approaches risk producing misleading or unreliable findings.

💡 Why It Matters

Automated text analysis offers a pathway to scale up empirical work on political language and conflict, but its scholarly value depends on careful validation and continued methodological innovation to avoid misinterpretation and misuse.