The Center for Social Data Analytics Colloquium speaker: Cassandra Tai

Google map

Toward Accountable AI: A Pipeline Framework for Evaluating Generative Models in Social Science

Abstract: We introduce a framework for evaluating generative AI (GenAI) in political and social science research, operationalized through a five-step Human–AI pipeline. The framework integrates GenAI’s multiple roles—as annotator, machine learning system, and “silicon participant”—and emphasizes transparency through dual-track metrics, cross-model robustness evaluation, human rationale auditing, and uncertainty and error correction. I demonstrate the framework using a study of climate stance detection among U.S. elected officials across Facebook and X, combining human domain expertise with GenAI’s efficiency in large-scale annotation. Through structured prompting, calibrated confidence thresholds, and cross-model auditing (GPT-5 and Llama-3.3), coupled with expert oversight, the pipeline demonstrates high inter-model reliability and near-human agreement while preserving interpretability. Together, the framework and application show how GenAI can be systematically evaluated and responsibly integrated into social-scientific workflows, advancing reproducibility, scalability, and accountability in AI-assisted research.

Bio: Cassandra Tai is an Assistant Research Professor and Assistant Director at the Center for Social Data Analytics (C-SoDA). Her research examines how artificial intelligence (AI) reshapes governance, communication, and public accountability. She develops and evaluates human–AI pipelines to analyze large-scale digital media data and assess the societal and institutional implications of emerging technologies, complemented by comparative public opinion studies. Her projects advance transparent, reproducible frameworks for integrating AI and computational methods into social science research. Her work has appeared or is forthcoming in journals such as American Political Science Review, Political Communication, Scientific Data, ACM WebSci 2025, and Political Science Research and Methods.