Questions, data sources, instruments.

<div><p>Background</p><p>A new undergraduate medical programme was instituted at the University of Botswana in 2009. In 2016, the Faculty of Medicine decided to conduct a comprehensive review of the programme. Participants at a planning workshop decided the review had to lead to an in-depth understanding of the programme. The challenge was to develop an evaluation process to achieve this aim.</p><p>Approach</p><p>Standards and theories used in other programme evaluations were investigated. A steering committee developed a ‘six-step process’ for the review of the programme as a complex system. The process used three evaluation models and various sources of evaluation standards to derive 90 evaluation questions. Quantitative and qualitative data were collected using 96 different instruments, with data triangulation prominently featured. Data analysis used an interpretivist approach. The review process was validated against the Cilliers’s ten features of a complex system.</p><p>Validation</p><p>The review process was validated against Cilliers’s ten features of a complex system. We found that the ‘six-step process’ illuminated each of these features in the MBBS programme in turn, and was therefore a valid way of evaluating this programme as a complex system.</p><p>Discussion</p><p>In the process of designing and using the ‘six-step process’ to evaluate a complex medical programme important lessons were learnt: starting the process with complexity theory at the forefront; being as inclusive as possible in data collection; also applying complexity theory to the evaluation of smaller programme components as ‘mini-complex systems’; and managing one’s inevitable insider bias.</p><p>Conclusion and utility</p><p>The ‘six-step process’ as it stands or in adaptation is likely to be useful in similar situations, where evaluators perceive the object of their evaluation to be a complex system, or a component of such a system.</p></div>