Benchmarking Best Practices for LLM-driven Automated Algorithm Design in Heuristic Optimization

Large Language Models (LLMs) are increasingly used to design optimization algorithms, in a paradigm referred to as LLM-driven Automated Algorithm Design (LLM4AAD). Such a shift challenges established benchmarking practice in optimization, which was built around evaluating a single fixed algorithm. In contrast, LLM4AAD systems produce a generated algorithm through a stochastic, LLM-driven generation process, and both the generation and the generated algorithms deserve attention in benchmarking. This position paper adapts and extends established benchmarking principles from evolutionary optimization to the specifics of LLM-driven algorithm design. We formalize the algorithm design problem, distinguishing the generated algorithm from the generation process that produces it, and argue that the two require distinct benchmarking targets and metrics. Using two representative examples, we systematically revisit the benchmarking goals established for classical meta-heuristic optimization, assess the current maturity of LLM4AAD practices against each, and introduce a further set of goals specific to the use of LLMs as the generative component. We further examine the reproducibility threats posed by stochastic, evolving, and often closed-source LLM backends, and outline minimum reporting practices to mitigate them. We conclude with a research agenda identifying the community infrastructure and evaluation protocols needed to establish shared benchmarking standards for the field.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-16
DOI
https://doi.org/10.5281/zenodo.24276211
Primary Topic
Metaheuristic Optimization Algorithms Research
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Benchmarking Best Practices for LLM-driven Automated Algorithm Design in Heuristic Optimization

Qingfu Zhang, Bas van Stein, Anna V. Kononova, Elizabeth Polgreen et al.
3 links
Zenodo (CERN European Organization for Nuclear Research)
Metaheuristic Optimization Algorithms Research
preprint

Benchmarking Best Practices for LLM-driven Automated Algorithm Design in Heuristic Optimization

Qingfu Zhang, Bas van Stein, Anna V. Kononova, Elizabeth Polgreen, Fei Liu, Urban Škvorc, Moritz Seiler, Francesca Da Ros, Nguyen Dang, Thomas Bäck, Rong Qu, Phong Le
preprint en

Abstract

Large Language Models (LLMs) are increasingly used to design optimization algorithms, in a paradigm referred to as LLM-driven Automated Algorithm Design (LLM4AAD). Such a shift challenges established benchmarking practice in optimization, which was built around evaluating a single fixed algorithm. In contrast, LLM4AAD systems produce a generated algorithm through a stochastic, LLM-driven generation process, and both the generation and the generated algorithms deserve attention in benchmarking. This position paper adapts and extends established benchmarking principles from evolutionary optimization to the specifics of LLM-driven algorithm design. We formalize the algorithm design problem, distinguishing the generated algorithm from the generation process that produces it, and argue that the two require distinct benchmarking targets and metrics. Using two representative examples, we systematically revisit the benchmarking goals established for classical meta-heuristic optimization, assess the current maturity of LLM4AAD practices against each, and introduce a further set of goals specific to the use of LLMs as the generative component. We further examine the reproducibility threats posed by stochastic, evolving, and often closed-source LLM backends, and outline minimum reporting practices to mitigate them. We conclude with a research agenda identifying the community infrastructure and evaluation protocols needed to establish shared benchmarking standards for the field.

Zenodo (CERN European Organization for Nuclear Research)
Leiden University (NL), University of Nottingham (GB), University of St Andrews (GB), City University of Hong Kong (HK), University of Zurich (CH), Paderborn University (DE), Centro di Riferimento Oncologico (IT), University of Edinburgh (GB)
Metaheuristic Optimization Algorithms Research
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon