Computational Social Science Speaker Series: Dr. Kaiping Chen

Speaker: Dr. Kaiping Chen, University of Wisconsin-Madison

Title: How GPT-3 responds to different publics on climate change and Black Lives Matter: A critical appraisal of equity in conversational AI

Abstract: Autoregressive language models, which use deep learning to produce human-like texts, have become increasingly widespread. Such models are powering popular virtual assistants in areas like smart health, finance, and autonomous driving, and facilitating the production of creative writing in domains from the entertainment industry to science communities. Despite growing discussions of AI fairness across disciplines, there lacks systemic metrics to assess what equity means in dialogue systems and how to engage different populations in the assessment loop. In this talk, Dr. Kaiping Chen will draw from theories of deliberative democracy and science and technology studies to propose an analytical framework for evaluating equity in human-AI dialogues. Using this framework, Dr. Chen will introduce a recent algorithm auditing study her team has conducted to examine how GPT-3 responded to different subpopulations on crucial science and social issues: climate change and the Black Lives Matter (BLM) movement. In the study, Dr. Chen and her collaborators built a user interface to let diverse participants have conversations with GPT-3. The study found a substantively worse user experience with GPT-3 among the opinion and the education minority subpopulations; however, these two groups achieved the largest knowledge gain, changing attitudes toward supporting BLM and climate change efforts after the chat. In this study, Chen’s team also traced these user experience divides to conversational differences and found that GPT-3 used more negative expressions when it responded to the education and opinion minority groups, compared to its responses to the majority groups. To what extent GPT-3 uses justification when responding to the minority groups is contingent on the issue. Dr. Chen will discuss the implications of the findings for a deliberative conversational AI system that centralizes diversity, equity, and inclusion.

Bio: Dr. Kaiping Chen (PhD, Stanford University) is currently an Assistant Professor in Computational Communication from the Department of Life Sciences Communication at the University of Wisconsin-Madison. Dr. Chen is also faculty affiliate at Department of Political Science, the Nelson Institute for Environmental Studies, and the Robert F. and Jean E. Holtz Center for Science and Technology Studies. Dr. Chen’s research use data science and machine learning methods as well as interviews to study to what extent digital media and technologies hold politicians accountable for public well-being and how deliberative designs can improve the quality of public discourse and mitigate misinformation. Dr. Chen’s work is interdisciplinary and draw from theories in communication, political science, and computer sciences. Chen’s works in utilizing big data tools and community engagement methods have been supported by the US National Science Foundation, Chan Zuckerberg Initiative, and American Family Insurance. Her works were published in flagship journals across disciplines, including the American Political Science Review, Journal of Communication, Journal of Computer-Mediated Communication, New Media & Society, Public Opinion Quarterly, Public Understanding of Science, Harvard Kennedy School Misinformation Review, International Public Management Journal, Proceedings of the National Academy of Sciences (PNAS), among other peer-review journals. Dr. Chen is also a civic engagement practitioner, with her continued passion to help local governments and communities in US and China implement and analyze innovative practices of engaging citizens throughout policymaking. For information about her work, you can visit: https://www.kaipingchen.com.