Retrieval-Based Evaluation of Cell Painting Feature Spaces Reveals Differences in the Preservation of Biologically Meaningful Phenotypic Similarity

Cell Painting enables high-dimensional phenotypic profiling of cellular states, but retrieval-based interpretation depends on whether the chosen feature space preserves task-relevant biological relationships. With pretrained Cell Painting feature extractors increasingly available, feature spaces should be qualified before downstream biological retrieval. Here, we developed a task-aware workflow for evaluating Cell Painting feature spaces on a curated U2OS JUMP-MOA reference plate. Three pretrained models, CellPaintSSL, OpenPhenom, and uniDINO, were applied in a zero-shot setting to the same image set, and the resulting profiles were analyzed using a copairs-based mean average precision (mAP) retrieval framework. We assessed compound-induced activity relative to dimethyl sulfoxide (DMSO) controls, same-compound profile resolution among active perturbations, and mechanism-of-action (MOA) annotation recovery at the compound-profile level. All three feature spaces showed strong prerequisite performance, with a large proportion of compounds passing both activity and distinctiveness criteria. However, MOA annotation recovery was partial and model-dependent. Although the overall number of recovered MOA annotations was similar across feature spaces, the specific MOA annotations recovered by each model differed. These results show that prerequisite profile quality does not guarantee recovery of the biological relationship being tested, such as shared MOA annotation, and support task-aware qualification of feature spaces before downstream interpretation.