bringing in the state for experiments and development efforts — when? how?

there have been a lot of mentions of governments, experiments, ownership, & development in the past two weeks that sparked a few ideas in my head. the underlying theme is that we need to start considering not just the political economy of the contexts in which we work but how to actually bring political and economic considerations – and interests in long-term sustainability, accountability, and ‘ownership’ – into program design and implementation, as well as into the experiments to test those programs. i first consider lessons from a totally hypothetical RCT. then, two quick reviews of new programs related to public sector involvement in development efforts.

first, the political economy of experimentation: lessons stemming from a totally hypothetical nation-wide RCT run in conjunction with the government of a totally hypothetical state.

as suggested before, experiments, should often be designed with the (likely) ultimate implementer in mind – ideally, in consultation with them. because much of development deals with public goods and market failures, there is a good chance the state would ultimately be involved in an experiment-born or experiment-tested program/policy in order to bring it to scale, institutionalize it, & sustain it.

this requires experimental design to start to look more like program design. research design cannot substitute for program or policy design. program & policy design requires inputs from the local context and from the field of study & practice relevant to the content of the intervention.

these designs must also account for politics. working with the government – at a large and visible scale – means what once may have been a more ‘neutral’ or ‘local’ experiment is now political. who gets access to the intervention, when, and how (Lasswell, 1936) becomes critical. this will often be inherently at odds with randomized evaluation (King et al, 2007) (Fox & Reich, forthcoming). it’s not impossible, it just takes a good deal of planning & savvy in both program and research design & implementation.

even in the face of political, technical, and social challenges to implementation of the program & the study, there will always be important lessons to be learned — but only if solid monitoring is taking place and process evals are reported. all stakeholders need to be on board with that up front (tough!).

• when does the government come in?
  • option 1: the experiment starts because of a program/policy the government wants to try or tweak. thus, it is government- or implementer-initiated. these experiments are, i believe, along the lines of what David Brooks suggests here.
  • option 2: researchers/technocrats have an idea that they want to try at a large-scale, and therefore approach the government. the gov/implementer should be on board before the design is finalized. they will have a much clearer idea than almost any researcher of what is politically feasible and what incentives will need to be offered to get the buy-in of on-the-ground implementers and bureaucratic structures. that is, the program design needs to be rigorous, with technical and theoretical aspects adjusted to the local political economy and an understanding of different stakeholders’ interests and capacity.
  • in either case: if the implementer seems unwilling to scale-up an idea in the design phase, the experiment should be re-considered and re-designed. if the government lacks capacity or is stretched too thin to scale-up and sustain the idea, then the extent to which capacity-building can and should be built into the experiment needs to be strongly considered (as well as whether an alternative approach is needed).

• how can the state & other stakeholders be brought in?
  • a wide variety of stakeholders should be involved in deciding how to measure an acceptable level of progress and ‘success.’ all stakeholders (including politicians, bureaucrats) should articulate what will make the experiment worthwhile to them and what they would need to see in order to ‘be convinced’ that the program/policy is worth continuing to pursue. treatment effect will be far from the only thing that matters.
  • responsibility, political risk, & political timelines need to be discussed explicitly. having technocrats and researchers involved may shield politicians from some blame if the program does not work as intended – but a plan to laud politicians for engaging in evidence-building policy and to allow them to take some credit when things go well – will be important (Fox & Reich, forthcoming).

• other lessons that could be learned from this hypothetical study:
  • for a variety of appearance & funding reasons – because of commitment to public sector involvement – we may want it to appear that the government is running the experimental program. but if it’s not what’s happening on the ground, re-assess and either build capacity or adapt the program to reality.
  • proof-of-concept experiments will still have a role in helping certain ideas seem less risky to stakeholders – because large-scale policy changes are risky, both in terms of domestic politics and global politics. these small-scale experiments are also the best testing ground for multiple treatment arms, rather than proliferating treatment arms when working at national-scale. however, even in these experiments, we should consider what outcomes will be politically relevant and what technical kinks we encounter, so that: (1) the results are more easily ‘sell-able’ to politicians and other implementers down the line and (2) it will be clearer what capacities and resources are necessary for the program/policy in question.

moving on from our hypothetical example (whew!)… on to two non-research-based examples of designing and implementing development efforts in ways to encourage state involvement and to increase its accountability & sustainability. what lessons about incentives & partnering could the experimental world – with an increasing eye toward scale-up and sustainability – learn from these?

the Gates Foundation has an interesting (and uncharacteristic) initiative on the table to get state governments in Nigeria more involved in vaccination and MDG efforts (h/t @KarenGrepin). this effort recognizes the key role of governmental leadership in the implementation of development programs (though these grants will go to the ‘implementing partner;’ states will be able to demonstrate ownership in part through co-funding awards projects (?)).

there are 11 process & outcome criteria on which programs will be judged, with awards going to the highest-performing state (also for the most-improved) in each of Nigeria’s geo-political zones. ultimately, it is not clear to me exactly how this will reward political effort or encourage the design of programs that can be institutionalized (vaccines – you have to keep giving them year after year!) – but kind of a cool idea to have all those government- & partner-designed pilots running simultaneously and rewarding some outcomes. it would be great to know what other sorts of evaluation will be going on outside of the 11 award criteria, so that we may all learn from challenges and successes.

meanwhile, Water for People (@NedBreslin) has an interesting commentary (h/t How Matters) on sustainability and ownership. NB suggests that fiscal and operational capacity and discipline of the implementers need to be central considerations in program design and implementation. NB suggests that programs and capital investments need to be designed and tailored to the demonstrated ability-to-pay of the community in question. that is, co-financing between a development agency (including researchers?), communities, and local government is required before new infrastructure was installed. the general idea seems to be this: ‘we (researchers(?)/philathropists/etc) want to work towards achieving X outcome in your [geographic area]; achieving X outcomes requires sustained effort and investment. we have a mix-and-match menu of ways to put together a program to work towards X goal; we should determine which option will be best for this community based on, in part, your ability to invest some money up front, to the extent that your payments will be required to continue to achieve X over the long-term.’ NB goes on to say, “free projects facilitate corruption… [gov] funding starts to be allocated by governments if NGOs (researchers?) use their finances as leverage, in financial partnership with host-country governments, rather than absolving them of their financial and developmental responsibilities.”

as experiments become more like programs (and programs more like experiments), and take place at larger scales and in greater collaboration with the public sector, we need to think carefully about what is required for sound program design and implementation, such that we can still have impact after an experiment ends (though, of course, long-term measurement of process & outcomes should be supported).