How Developers Discuss Generative AI Across the AI-Integration Gradient: A Longitudinal, Multi-Repository Study of GitHub Issues

Abstract How open-source communities discuss generative artificial intelligence(AI) remains unclear. We study generative-AIdiscussion in GitHub Issues from January 2021 to August 2026 in sevenrepositories spanning an AI-integration gradient : Visual Studio Code(VS Code), where AI features are prominent; Home Assistant, where AI is one feature;and five projects without built-in AI features. Two annotators independently labeled a stratified sample of 2{,}080from 52{,}598 keyword-matched issues with large language model assistance; we add topic modeling,interrupted time series, and survival analysis. AI relevance agreement ishigh in repositories with AI features (Cohen's \((\kappa=0.78)\)--\((0.85)\)) but near zero in CPythonand Rust (\((\kappa\leq0.08)\)), where most disagreements involve issues whose AI evidenceappears only in the comment thread, typically a maintainer invoking anAI-contribution policy. We therefore report every estimate under bothannotators. Consistently across annotators, AI-feature trackers carry operationaldiscourse; keyword precision falls from 53--61% in VS Code to under 3% inFlutter and Godot; and among issues both annotators deem AI-relevant,AI-generated contributions make up 8 of 20 in the bystander projects but 1of 293 in VS Code. Their prevalence, however, depends on the annotator(CPython: 0.6--16%). Issue volume increased in early 2025, but estimates associated withagent-mode releases depended on breakpoint choice; the largest discontinuitycoincided with consolidation of Copilot Chat issue tracking. AI-related issues close faster inthe AI-feature repositories (hazard ratio 1.4) but not consistentlyelsewhere. The relevance classifier has lower precision outside VS Code.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-13
DOI
https://doi.org/10.21203/rs.3.rs-11214426/v1
Primary Topic
Software Engineering Research
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

How Developers Discuss Generative AI Across the AI-Integration Gradient: A Longitudinal, Multi-Repository Study of GitHub Issues

Akito Monden, Peerapas Chinnapatpakdeekul, Panida Rumriankit
1 links
Research Square
Software Engineering Research
preprint

How Developers Discuss Generative AI Across the AI-Integration Gradient: A Longitudinal, Multi-Repository Study of GitHub Issues

Akito Monden, Peerapas Chinnapatpakdeekul, Panida Rumriankit
preprint en

Abstract

Abstract How open-source communities discuss generative artificial intelligence(AI) remains unclear. We study generative-AIdiscussion in GitHub Issues from January 2021 to August 2026 in sevenrepositories spanning an AI-integration gradient : Visual Studio Code(VS Code), where AI features are prominent; Home Assistant, where AI is one feature;and five projects without built-in AI features. Two annotators independently labeled a stratified sample of 2{,}080from 52{,}598 keyword-matched issues with large language model assistance; we add topic modeling,interrupted time series, and survival analysis. AI relevance agreement ishigh in repositories with AI features (Cohen's \((\kappa=0.78)\)--\((0.85)\)) but near zero in CPythonand Rust (\((\kappa\leq0.08)\)), where most disagreements involve issues whose AI evidenceappears only in the comment thread, typically a maintainer invoking anAI-contribution policy. We therefore report every estimate under bothannotators. Consistently across annotators, AI-feature trackers carry operationaldiscourse; keyword precision falls from 53--61% in VS Code to under 3% inFlutter and Godot; and among issues both annotators deem AI-relevant,AI-generated contributions make up 8 of 20 in the bystander projects but 1of 293 in VS Code. Their prevalence, however, depends on the annotator(CPython: 0.6--16%). Issue volume increased in early 2025, but estimates associated withagent-mode releases depended on breakpoint choice; the largest discontinuitycoincided with consolidation of Copilot Chat issue tracking. AI-related issues close faster inthe AI-feature repositories (hazard ratio 1.4) but not consistentlyelsewhere. The relevance classifier has lower precision outside VS Code.

Research Square
Okayama University (JP), Kasetsart University (TH)
Software Engineering Research
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon