Evaluation of Programs with Multiple Objectives: Multidimensional Methods and Empirical Application to Progresa in Mexico

Abstract

Development programs and policy interventions frequently have multiple simultaneous objectives. Existing quantitative evaluation approaches fail to fully accommodate this multiplicity of objectives. In this paper we adapt the multidimensional poverty measurement approach developed by Alkire and Foster (2011) to the estimation of treatment effects for programs with multiple objectives. We use the potential outcomes framework to show that differences in Alkire-Foster indices between treated and control samples correspond to average treatment effects estimates of outcomes of interest under experimental conditions, and develop further methods of analysis to explore these multidimensional treatment effects. We discuss issues of index design encountered in practice and provide an illustrative example. We apply the methods developed to evaluate the conditional cash transfer program Progresa in Mexico, finding significant multidimensional effects of the program. Further analysis shows that these treatment effects are driven mainly by impacts on school attendance and health visits, objectives that correspond directly to the conditions of the program. There is no evidence for heterogeneity of the treatment effects by the extent to which the beneficiary failed to achieve the objectives at baseline. This study complements the extensive literature on the evaluation of Progresa and other development programs, comprising studies that focus on particular objectives or outcomes of the program. We hope that the methods developed here will find wide application to the evaluation of programs with multiple objectives.

Development programs and policy interventions frequently have multiple simultaneous objectives. Existing quantitative evaluation approaches fail to fully accommodate this multiplicity of objectives. In this paper we adapt the multidimensional poverty measurement approach developed by Alkire and Foster (2011) to the estimation of treatment effects for programs with multiple objectives. We use the potential outcomes framework to show that differences in Alkire-Foster indices between treated and control samples correspond to average treatment effects estimates of outcomes of interest under experimental conditions, and develop further methods of analysis to explore these multidimensional treatment effects. We discuss issues of index design encountered in practice and provide an illustrative example. We apply the methods developed to evaluate the conditional cash transfer program Progresa in Mexico, finding significant multidimensional effects of the program. Further analysis shows that these treatment effects are driven mainly by impacts on school attendance and health visits, objectives that correspond directly to the conditions of the program. There is no evidence for heterogeneity of the treatment effects by the extent to which the beneficiary failed to achieve the objectives at baseline. This study complements the extensive literature on the evaluation of Progresa and other development programs, comprising studies that focus on particular objectives or outcomes of the program. We hope that the methods developed here will find wide application to the evaluation of programs with multiple objectives.