When Can We Trust an Automated Phase Identification?

Phase identification is a foundational step in powder X-ray diffraction (XRD), because the identified phases usually define the model used for the quantitative analysis that follows. An automated result is only valuable if it can be trusted. Trust depends on which phases are considered, how well real measurement variability is represented, how the present/absent decision is made, and how the result is verified. We examine these questions using seven real laboratory scans from a commercial iron-ore quality-control package. The scans are real laboratory measurements covering oxidic and pyritic ore types and a technical mill-scale product. The package defines 28 phases and provides their crystal structures and reference Rietveld results. The same structure files feed three paths: the reference models, the synthetic training data for a convolutional neural network (CNN), and a small DIFFRAC.EVA user database. The CNN is the published network of Schuetzke et al. (2021); its per-sample results are drawn from the associated master’s thesis (Schuetzke, 2019). Using these same scans, the automatic search/match workflow is evaluated in this work, allowing a like-for-like comparison against that network on the same scans. For the non-trace phases, first-pass automatic DIFFRAC.EVA reaches 66% recall, 100% precision, and an F1 score of 79%. The CNN reaches 90% recall, 79% precision, and an F1 score of 84%. While the CNN recovers more phases it also proposes more that are absent from the reference model. A highly informative case study is Wüstite. The small user database holds only one Wüstite entry, whose lattice parameter does not match the measured material, and DIFFRAC.EVA misses the phase. Expanding the database with lattice-parameter variants, or widening the match tolerance, recovers it; the CNN identifies the phase because comparable variation was already present in its training set. The results indicate that credibility depends less on the choice between a neural network and classical search/match than on how well physical variability is represented, how uncertainty is handled, and how the result is verified. This has direct consequences for self-driving laboratories, where every automated decision inherits the strengths and limits of the analysis that feeds it. The design of the search space is identified as a distinct question for future work rather than optimized here.