Addressing selection bias while estimating aggregate development effectiveness: can we obtain externally valid estimates at portfolio level?

As a result of greater demand from funders and governing bodies to assess their ‘impact for money’, the need for aggregate aid effectiveness for development agencies has become increasingly prominent. Measurement of aid effectiveness or corporate impact, however, requires attribution, i.e. the capacity to causally attribute observed impacts from an investment project to the project alone, as well as counterfactual-based impact evaluations. As the prevalence of impact evaluations grows among development agencies, attributable aggregate development effectiveness to measure institution-wide results is a necessary component of any institution’s impact evaluation agenda to ensure accountability, identify performance gaps, and areas for improvement. ‘Intelligent’ aggregation additionally requires three key elements, namely a critical mass of impact evaluations representing the investment portfolio of the agency in question, a methodology for aggregation, and a universe of projects from where the projects evaluated were randomly drawn. In this paper, a novel methodology based on meta-analytic techniques, selection models and projection methods is proposed along with a number of systematic analyses that adjust for the possible presence of selection bias, a crucial factor to take into account while estimating aggregate development effectiveness.