Invited Paper: Rethinking Benchmarks for Parallel Machine Learning Techniques: Integrating Qualitative and Quantitative Evaluation Metrics

Growing model complexity and data volumes in Machine Learning (ML), especially Deep Learning (DL), necessitate parallel processing for efficient, scalable computation. Benchmarking is critical for evaluating parallel ML techniques. Current methodologies predominantly emphasize quantitative metrics like throughput and accuracy, exemplified by MLPerf and HPC-AI500. Such quantitative focus neglects key qualitative factors-portability, robustness, deployment complexity, and usefulness-essential for practical parallel ML. This paper argues for a paradigm shift in benchmarking, advocating for a holistic evaluation framework that inherently integrates qualitative assessments alongside traditional quantitative measures. To this end, we introduce NNoPP (Neural Network on-top-of Parallel Processing), a structured evaluation approach designed to complement existing benchmarks. NNoPP proposes criteria---novelty, portability, performance, scalability, complexity, usefulness, and sustainability---methodically defined for real-world ML solution viability, sustainability, and impact. Beyond performance numbers, these metrics promote sustainable solutions: computationally efficient, practically deployable, broadly applicable, and robust across diverse contexts. Through illustrative case studies across key DL model families, we demonstrate the application of NNoPP and its capacity to provide nuanced insights beyond conventional benchmark rankings. Future directions for parallel ML benchmarking emphasize integrated metrics, dynamic workloads, domain-specific relevance, and collaborative community evolution. Ultimately, this paper emphasizes a re-envisioned holistic benchmarking approach that fosters the adoption of parallel ML solutions that are both quantitatively superior and qualitatively robust, useful, and sustainably impactful for continuous advancement.