Probing the limits of mobile phone metadata for poverty prediction and impact evaluation

A series of recent papers demonstrate that mobile phone metadata can, together with machine learning, estimate the wealth of individual subscribers and accurately target cash transfer programs. In the context of an emergency cash transfer program in Haiti, we combine surveys and mobile phone call detail records (CDR) to test whether such methods can be used to estimate the program’s impact on household expenditures . We find that CDR-based predictions of total and food expenditures are much less accurate than predictions of wealth—particularly when estimated on a relatively homogeneous sample of rural communities eligible for the program. While impact estimates based on conventional survey data are positive and statistically significant, estimates based on CDR predictions are not statistically significant. In a postmortem discussion, we assess reasons for this failure and discuss the implications for using big data in poverty measurement and impact evaluation. • Mobile phone data and machine learning are better predictors of wealth than expenditure. • Prediction accuracy goes down when sampling from primarily poor households. • A tradeoff exists between data useful for program evaluation and for welfare prediction.