Feasibility and performance assessment of unannounced standardized patients method before full study of primary care quality assessment in China: a pilot study

Abstract Background The unannounced standardized patient (USP) method is recognized as the gold standard for healthcare quality assessment due to its high reliability and minimal recall bias. However, its high implementation cost makes pilot studies essential prior to large-scale field surveys, while evidence on the procedure and value of USP-based pilot studies remains scarce. This pilot study evaluated the feasibility and performance of the USP method for a nationwide large-scale primary healthcare (PHC) quality assessment across seven provinces in China. Methods A prospective, multicentre, non-randomized feasibility and methodological evaluation study was adopted throughout USP recruitment, interview, training, and field visit stages. Remote recruitment and interviews were conducted in seven provincial capitals of China, followed by hybrid online-offline training based on 11 common PHC cases. Qualified USP-investigator pairs completed unannounced visits to community health centers. Script accuracy and checklist completion consistency were analyzed to assess USP performance and data reliability. Results A total of 354 applicants (252 for USP, 102 for investigator) were recruited between July and August 2020, with 50 qualified USP-investigator pairs finally enrolled in field visits. Of 776 attempted visits, 729 were completed successfully, with a success rate of 93.94%. The average script accuracy was 81.25%, peaking at 86.75% at the fourth visit. Overall, 93.55% of USPs met qualification criteria (31.18% fully qualified, 62.37% partially qualified), and data collection methods showed moderate reliability overall, with notable variation across cases. Conclusions This pilot study demonstrates the feasibility of the USP recruitment, interview and training protocol, with trained USPs able to perform standardized role-playing for PHC quality assessment. However, several areas require refinement before the full study, including script fidelity, case-specific checklist agreement, and quality control procedures. The developed data collection tools, operational manual and experience from this pilot provide a foundation for the subsequent full-scale study.