SIMVALE: A Generalizable Simulator Validation Framework Combining Embedding Clustering and Feature-Based Metrics

The rise of Large Language Model (LLM)-powered simulators has enabled highly realistic modeling of complex social phenomena, significantly reducing the costs and efforts associated with real-world data collection. However, their reliability remains a persistent challenge, and existing validation approaches offer only partial generalization, typically evaluating simulator realism along isolated dimensions and at fixed levels of granularity. To this end, this thesis introduces SIMVALE (SIMulator VAlidation with Latent Embeddings) — a generalizable multi-dimensional validation framework enabling quantitative assessment of LLM-based simulators. SIMVALE leverages latent embedding clustering and yields both global and local feature-based metrics to evaluate how closely a simulator reflects the reality it aims to model, ultimately supporting its improvement. Furthermore, SIMVALE is tested on a real case study to assess the simulator’s ability to replicate general behavioral patterns, toxic profiles, and the effects of moderation interventions.