GGAgent: Green Coding Agent via Multi-Objective Configuration Optimization

Overview GGAgent (Green/Greener Agent) addresses the critical challenge of sustainable coding agent deployment by optimizing the trade-off between performance improvement and resource consumption. While coding agents can successfully solve complex real-world tasks, they consume significant computational resources—a single agent run can consume over 100,000 tokens, translating to hundreds of dollars per optimization task and substantial environmental impact. **The Problem**: Without careful tuning, coding agents can consume over 100,000 tokens per task, creating prohibitive operational costs and substantial environmental impact. The energy consumed by an optimization agent can be so substantial that the resulting code may need to be executed hundreds of thousands of times to reach an energetic "break-even point," making the optimization a net energy loss. **Our Solution**: GGAgent employs NSGA-II multi-objective optimization to automatically discover Pareto-optimal agent configurations across the vast hyperparameter space, enabling up to **43.6% runtime reduction** while maintaining or improving software quality. ### Research Contributions This framework demonstrates three key contributions: 1. **Multi-Objective Optimization Framework**: Discovers Pareto-optimal coding agent configurations achieving up to 43.6% runtime reduction (853.3s vs 1513.3s) while maintaining equivalent functionality, and up to 135× hypervolume improvement over default configurations. 2. **Comprehensive Hyperparameter Analysis**: Reveals that temperature is the most critical factor (-0.392 performance impact), while timeout constraints universally improve efficiency and top_p creates performance-runtime trade-offs. 3. **Actionable Green SBSE Strategies**: Provides scenario-specific optimization guidelines for runtime reduction, quality maximization, and context-specific deployment. Repository Structure ``` GGAgent/ ├── mini-swe-agent/ # Base coding agent framework ├── ga/ # Genetic algorithm optimization │ ├── ga.py # Main optimization script (NSGA-II) │ ├── agentsRunner.py # Agent evaluation runner │ ├── problem.py # Multi-objective problem definition │ ├── run_single_config.py # Single configuration evaluator │ ├── template.yaml # Configuration template │ ├── evaluation/ # Evaluation infrastructure │ └── results/ # Optimization and analysis results ├── configurations/ # Agent configuration files └── requirements.txt # Python dependencies ``` Installation ### 1. Environment Setup ```bash # Create conda environment conda env create -f environment.yml conda activate sweperf # Or install with pip pip install -r requirements.txt ``` ### 2. API Configuration Set up your Gemini API key: ```bash export GEMINI_API_KEY="your_api_key_here" ``` ### 3. Base LLM Configuration To change the base LLM, modify the model configuration in yaml files: ```yaml model: model_name: "gemini-v25-pro" # Change to desired model ``` Methodology ### Multi-Objective Optimization Framework GGAgent employs NSGA-II with population size 5 and 5 generations to explore the coding agent configuration space. For each candidate configuration, the system: 1. **Agent Evaluation**: Agent receives performance optimization tasks and generates code patches through iterative reasoning 2. **Resource Measurement**: Measures agent runtime consumption during the optimization process 3. **Quality Assessment**: Executes generated patches in isolated Docker environments to measure correctness and performance gain 4. **Pareto Analysis**: Optimizes three objectives (maximize correctness, maximize performance gain, minimize runtime) and outputs Pareto-optimal configurations ### Configuration Space The framework optimizes 8 hyperparameters across three categories: - **LLM Parameters**: Temperature, Top_p, Max_tokens - **Agent Parameters**: Step_limit, Cost_limit, Env_timeout, LLM_timeout - **Prompt Parameters**: Template variant selection ### Evaluation Benchmark Uses SWE-Perf benchmark focusing on the astropy project: - **Training Set**: 9 instances for NSGA-II optimization - **Validation Set**: 3 instances for performance validation - **Environment**: Isolated GCP server (4 CPUs, 16GB RAM, Ubuntu 25.04) - **Statistical Validation**: 20 measurements per instance with Mann-Whitney U test (p < 0.1) Usage ### Running Multi-Objective Optimization Execute the main optimization process: ```bash cd ga/ python ga.py ``` This will: - Run NSGA-II for 5 generations with 25 total evaluations - Generate configuration files in `configurations/` directory - Save detailed results and analysis in `ga/results/` - Identify Pareto-optimal solutions ### Testing Single Configurations Evaluate individual configurations: ```bash python run_single_config.py --config path/to/config.yaml ``` Results and Analysis ### Optimization Results - **Configuration Files**: `configurations/` - Generated YAML files for all evaluated configurations - **Raw Results**: `ga/results/optimization_results_20250919_154742.csv` - Complete evaluation data - **Pareto Front**: `ga/results/pareto_front_20250919.csv` - Identified optimal solutions ### Default vs Pareto-Optimal Configurations Based on our optimization results, we identified 5 Pareto-optimal configurations: | Configuration | Runtime (s) | Performance (%) | Correctness | Key Features | |---------------|-------------|-----------------|-------------|--------------| | Default | 1513.3 | 0.00 | 2.0 | Baseline configuration | | Config #4 | 943.1 | 0.00 | 4.0 | Speed-focused (37% faster) | | Config #5 | 984.8 | 6.43 | 8.0 | Quality-focused (4× correctness) | | Config #9 | 958.1 | 0.00 | 7.0 | Balanced performance | | Config #15 | 1400.1 | 10.67 | 7.0 | Performance-optimized | | Config #16 | 853.3 | 0.00 | 0.0 | Maximum speed (43.6% reduction) | **Configuration Files Location:** - Default: `configurations/default.yaml` - Pareto configs: `configurations/pareto_eval{4,5,9,15,16}_config.yaml` ### Validation Results Validation experiments on held-out instances: - **Validation Data**: `ga/results/validation_*_summary.csv` - Performance on unseen tasks - **Hypervolume Analysis**: Confirms optimization effectiveness across different problem instances ### Analysis Scripts Located in `ga/results/`: - **`RQ1_pareto_analysis.py`**: Pareto front analysis and hypervolume calculations - **`RQ1_validation_hypervolume.py`**: Validation hypervolume computation - **`RQ2_hyperparameter_analysis.py`**: Random Forest feature importance analysis