Question form Matters: Examining Trust in Government Through Open and Closed Survey Items

Abstract How we ask questions in surveys significantly impacts the answers. While previous research has examined differences between open-ended and closed-ended survey responses, our understanding remains fragmented, especially considering the growing interest in open-ended questions spurred by advances in automated text analysis. We explore how computational text analysis methods––particularly large language models––can efficiently analyze open-ended responses with promising sentiment and actor identification results, though topic classification remains more challenging. Then, we systematically compare open-ended and closed-ended responses on trust in the Austrian coalition government to assess how different formats shape the expression of political trust. We find significant discrepancies between the trust levels indicated in closed-ended questions and the trust described in open-ended responses, with paired t-tests also revealing statistically significant differences. These findings highlight that question format affects how respondents express their views, raising important considerations about the usefulness and validity of trust measures. They highlight the unique value of open-ended questions in capturing nuanced perspectives often missed by closed-ended formats. This has broader implications for survey design: including open-ended questions could enrich data quality and provide deeper insights into public opinion.