Can LLMs Write CI? A Study on Automatic Generation of GitHub Actions Configurations (Replication Package)

This repository contains the replication package for the paper "Can LLMs Write CI? A Study on Automatic Generation of GitHub Actions Configurations," accepted at the 41st IEEE International Conference on Software Maintenance and Evolution 2025 (ICSME'25). The package provides all resources needed to reproduce the experiments and results presented in the paper. How to Cite If you use this package, please cite our paper: Taher A. Ghaleb and Dulina Rathnayake. "Can LLMs Write CI? A Study on Automatic Generation of GitHub Actions Configurations." In Proceedings of the 41st IEEE International Conference on Software Maintenance and Evolution (ICSME), 2025. @inproceedings{ghaleb2025llm4ci, title={Can LLMs Write CI? A Study on Automatic Generation of GitHub Actions Configurations}, author={Ghaleb, Taher A. and Rathnayake, Dulina}, booktitle={Proceedings of the 41st IEEE International Conference on Software Maintenance and Evolution (ICSME)}, year={2025} organization={IEEE} } Package Structure project-root/ ├── data/ # Datasets and LLM outputs used in the study │ ├── GitHubActions_Docs.csv # Documentation data for GitHub Actions │ ├── models_contexts.csv # Contexts provided to LLMs │ ├── GitHubActions_gemma3-12b_output.csv # Output from Gemma 3-12b │ ├── GitHubActions_codegemma-7b_output.csv # Output from CodeGemma 7b │ ├── GitHubActions_gpt-4o_output.csv # Output from GPT-4o │ ├── GitHubActions_codellama-7b_output.csv # Output from CodeLlama 7b │ ├── GitHubActions_gpt-4.1_output.csv # Output from GPT-4.1 │ └── GitHubActions_llama3.1-8b_output.csv # Output from Llama 3.1-8b ├── scripts/ # Scripts for data collection, LLM prompting, and analysis │ ├── ci_docs_selenium_crawler.py # Crawls and collects CI documentation using Selenium │ ├── prompt_llms.py # Prompts LLMs and collects their outputs │ └── similarity_calculation_and_analysis.py # Calculates similarity scores and performs analysis ├── results/ # Experiment outputs and analysis results │ ├── GitHubActions_Similarity_Scores_Six_LLMs.csv # Similarity scores for six LLMs │ └── GitHubActions_Similarity_All_LLMs_boxplot.pdf # Boxplot visualization of similarity scores ├── requirements.txt # Python dependencies └── README.md # Project documentation Installation This package was developed and tested with Python 3.13.2. Clone the repository and install the required dependencies: pip install -r requirements.txt Additional Requirements Chromedriver: Required for running ci_docs_selenium_crawler.py. Please download Chromedriver and ensure it is in your system PATH or specify its location in the script. Ollama Framework: Required for running LLMs via the ollama Python package. See Ollama documentation for installation and setup instructions. OpenAI API Key: Required for using OpenAI models in prompt_llms.py. See OpenAI's Developer quickstart for more details about using the OpenAI API. You need to create an account on the OpenAI API platform, generate an API key, and then set it as an environment variable, as follows: export OPENAI_API_KEY=your_api_key_here (for Linux and Mac), orset OPENAI_API_KEY=your_api_key_here (for Windows) Usage Collect CI Documentation: This script accepts documentation URLs for various CI services, but was tested on GitHub Actions Documentation. Script: python scripts/ci_docs_selenium_crawler.py Prompt LLMs and collect outputs: This script supports running a variety of LLMs. In our experiments, we evaluated two OpenAI models and four Ollama-supported open-source models, but you may use it with other compatible models as well. For Ollama models, ensure the Ollama framework is running. Then, pull the required open-source models listed in data/models_contexts.csv before running the scripts. For each model name in the CSV, run: ollama pull <model_name> For OpenAI models, ensure your API key is set. Script: python scripts/prompt_llms.py All models' outputs will be saved in the data/ directory Calculate similarity and analyze results: This script computes similarity scores between LLM-generated and reference GitHub Actions configurations using multiple metrics (Cosine Similarity, Euclidean Distance, Tree Edit Distance, ROUGE-L, and chrF). It also performs correlation analysis and statistical significance testing to compare model performance, and generates summary tables and visualizations (such as boxplots) for further analysis. Script: python scripts/similarity_calculation_and_analysis.py All outputs and analysis results will be saved in the results/ directory. Data The data/ folder contains all datasets and LLM outputs used in the study. See the paper and script comments for details on each file. Results The results/ folder contains: Similarity scores for LLM-generated GitHub Actions configurations. Visualizations and summary statistics as reported in the paper. Docker Setup If you want to run the experiments on a standalone image using Docker, follow these steps: 1. Install Docker Download and install Docker for your platform from the official website: https://www.docker.com/products/docker-desktop. Follow the instructions for your operating system. 2. Save the Dockerfile Create a file named Dockerfile (no extension) in the project root containing the following instructions. You can refer to the associated GitHub repository (https://github.com/Taher-Ghaleb/ICSME25-LLM4CI) for any updates, including the ready-to-go dockerfile. ────────────────────────────────────────────────────────────────────── # Use Python 3.13 base FROM python:3.13-slim # Install system dependencies including build tools for packages with C extensions RUN apt-get update && apt-get install -y \ build-essential libffi-dev libssl-dev \ wget curl unzip \ python3-dev chromium chromium-driver \ && rm -rf /var/lib/apt/lists/* # Upgrade pip, setuptools, wheel RUN pip install --upgrade pip setuptools wheel # Set working directory WORKDIR /project # Copy requirements and install COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt # Copy project files COPY . . # Make chromedriver available in the current working directory RUN ln -s /usr/bin/chromedriver ./chromedriver # Install Ollama CLI RUN curl -s https://ollama.com/install.sh | bash # Set environment variable for OpenAI API Key ENV OPENAI_API_KEY="" # Default command to run when the container starts CMD ollama start & bash ────────────────────────────────────────────────────────────────────── 3. Build the Docker Image From the project root, build the Docker image: docker build -t llm4ci-project . 4. Run the Docker Image Now you can run the Docker image and provide your OpenAI API key, as follows: docker run -it --rm -e OPENAI_API_KEY=your_api_key_here llm4ci-project 5. Execute the steps in the Usage section above License Code in this repository is licensed under the MIT License. See the LICENSE file. Data files in this repository are licensed under the Creative Commons Attribution 4.0 International (CC BY 4.0) license unless otherwise noted. You are free to share and adapt the data with appropriate credit.