Center and Peripherary in Doing Development Differently

I have spent almost three weeks back in TX, which was supposed to be, in part, a time of immense productivity in front of our fireplace (yes, it is chilly here. Probably not enough to warrant a fire but still. I am sitting in front of the fireplace and paying for carbon credits to mitigate the guilt.) I brought home big batches of reading but am taking back far more of it with me to Delhi than I had planned.

Nevertheless, I did finally make it through Duncan Green’s post on his immediate thoughts on Doing Development Differently from Matt Andrews and team. So, that’s only three months behind schedule.

Many things are, of course, striking and exciting about this movement, including the idea of rapid iterations to promote (experiential) learning and tweaks, the importance of morale and relationships, and the time horizon.

But the most striking thing had to do with immersion, deep study and deep play*.

deep study of the system, based on continuous observation and listening. In Nicaragua, UNICEF sent public officials out to try and access the public services they were administering, and even made the men carry 30lb backpacks to experience what it’s like being pregnant! This is all about immersion, rather than the traditional ‘fly in, fly out’ consultant culture.

The idea is, it seems, to strike a blow at the ‘consultant culture’ of folks from D.C., London and Geneva parachuting in to solve problems (there’s probably an interesting discussion to be had about the relevance of area studies in this approach). But that is for another time. What is most immediately striking is that Duncan doesn’t report on UNICEF folks making consultants visiting Nicaragua from NYC head out to remote areas and try to access services with pregnant-backpacks.

If I read the anecdote correctly (is there more written about this somewhere?), the target was public officials, which I take to mean Nicaraguan civil servants and politicians based in the capital or another metropolis. Which is an important (re-)lesson. Being from X country doesn’t automatically make you knowledgeable about all areas and details of X country (duh). Probably many of us have sat with civil servants who talk about ‘the hinterlands’ and ‘backwards’ areas and who seem quite surprised at what they find there, if they visit at all. There is a vast difference between the high-level and the street-level, between big decisions about adopting and championing a policy and the many small decisions involved in implementing that idea. Implementation is, as always, profoundly local. (This idea, incidentally, also applies to study design and the relationships between PIs, their research assistants and the field teams.)

This all suggests that, maybe, doing development differently (and probably doing evaluation differently) also has to do with shifting ideas about center and periphery (globally as well as nationally), about who has relevant knowledge, and thinking about immersion for program designers and decision-makers of a variety of types, whether from the country in question or not. This, in part, raises questions about who is doing the iteration and learning and how lessons are passed up as well as down different hierarchies (and spread horizontally). looking forward to hearing and thinking more.

*It’s hard to resist a Geertz reference, since ‘continual observation and listening’ sounds an awful lot like ‘participant-observation,’ a study technique that almost *never* comes up in “mixed-methods’ evaluation proposals.

causes, explanations, & getting stuff done

@edwardcarr, i also have a confession, which is that i have a small crush on you right now for the post in which you make a confession about causality and try to disentangle causes, mechanisms, and information that can be used to understand, revamp, and scale programs. i‘d like to try to tweak the argument, especially in light of the recent excitement about the publication of null results from a bombay-based cluster-randomized trial on health and pregnancy, described here.

the tweak is to separate out two categories of information that are important and may be better gleaned from qualitative inquiry and analysis, though more adaptive/iterative and process-focused quantitative data can also play a role. i think the general mindset on what constitutes (and who has) useful and useable information is as important as the difference between quant or qual analysis.

1. how the program actually brought about an effect. this point is the focus of ed’s post, as well as levy paluck’s nice paper on qualitative methods and field experiments, which distinguishes between causal effects and causal mechanisms. mechanisms can, to some degree, be pursued and examined through ever-proliferating treatment arms in RCTs… but observation and purposive, systematic conversation are very helpful. if i read ed carr’s piece correctly, he wants more (data collected with an eye towards) explanation in order to pursue (a) a deeper understanding – beyond ‘story time’ – of what moderates and mediates the (potentially causal) relationship between X and Y within the study context, to also help us gain (b) a deeper understanding of the external validity of the findings, which could inform adaptation and replication. both are important goals for studies with any intention of scaling.

2. how the program was experienced by a range of stakeholders and how it could have been done better. this part doesn’t feature in ed’s post or levy paluck’s piece but is important. sometimes i feel like when i talk about process, everyone breaks out in log-frame hives. take a deep breath. i don’t just mean process checklists and indicators. i mean recording how things went, deviations from the study design, and seeking feedback from study participants, study facilitators, study staff, and other study stakeholders. in the bombay experiment referenced above, the team had a process evaluation officer, who consistently surveyed staff and documented  meetings with the participants. these data allowed the researchers to know, among other things, that the participating urban women “balked” at collective action but were happy to share information one-on-one — a fairly useful finding for anyone else designing a program with similar goals or in a similar population. i think the researchers could have gone slightly further in asking participants about possible explanations for the similarity of outcomes in the treatment and comparison group — but the centrality of process evaluation is clear nevertheless. in a similar vein, campos et al draw lessons from experiments that didn’t happen and propose that researchers need to “work more on delivery and better incentivize project staff.” this, too, suggests a need to better collect (and use) information on delivery process and staff perceptions of projects, which means making time (and setting aside money) to solicit this information and finding ways to incorporate it into study findings. it also means taking program design as seriously as experimental design.

in sum, explanation matters. collect data that allows for better explanation of the mechanisms underlying causal effects as well as the process by which those mechanisms were put in place. in the meantime, everyone needs to do more work on figuring out how to present these types of data in a way that is easily accessible to and valued by a variety of researchers and practitioners.

p4p, habits, context, process

Mark Nichter and Prascilla Magrath (hereafter, M&N) have a nice new paper on pay-for-performance (P4P), stating that “understanding the processes by which P4P targets are reached demands a reorientation… towards an understanding of motivation as a component within a complex adaptive social system.” (Complex systems are so hot right now!)

the paper is worth reading in full. the authors also raise important points that i have wanted to address for some time and hopefully will do so in more depth in the future. these relate to process; motivation & context; and experiments.

1. a key question that is not always sufficiently asked or answered in health systems – or other – research is ‘how/why did X intervention not/work in Y setting?” this is an issue of both internal validity / credibility and external validity. internally, answering these questions means looking at the intervention from the perspective of multiple stakeholders to get a nuanced, full view of how or why something worked. externally, of course, the more the context and processes are understood, the easier it may be to predict when and where else the intervention may be successful.

answering these types of questions involve qualitative as well as quantitative work and looking at implementation process as well as impacts. hopefully i will have a write-up on studying implementation in the near future with @jonathan_payne [yes, Jon, I am putting it in writing as a commitment device]. the call for these types of work in the study of interventions – including but not limited to explicitly experimental approaches – include papers by Paluck, Mills, and Ssengooba, as well as the M&N paper discussed here.

i want to quickly make amends for the past times i have denigrated process indicators. it’s true that process indicators can become as meaningless as they are made into check boxes of routines (such as number of posters hung or meetings held). also, process indicators without impact indicators don’t get us as far as we need to go in saying whether something worked or should be done again. but, recording and evaluating process is necessary, if not sufficient, in saying how something worked and whether it could work again (and where).

2. M & N draw on Bordieu to provide a frame for studying the implementation and impacts of P4P – but the point is more widely applicable. M & N draw on the concept of ‘habits’ – attitudes, dispositions, actions – of individuals and social groups to help explain how interventions are received differently in different places. this has interesting parallels with Stein‘s work on ‘habits,’ for which he draws on the work of Velben; these parallels and lessons deserve more exploration than i give them here.

both roughly aim at the idea of shaping interventions to a given contextual reality, including political, economic, and administrative structures but also social processes and group, sub-group, and individual dispositions and ‘habits of thought.’

both also suggest the need to consider how an intervention’s positive & negative effects will ripple out in both time and space, as well as how positive effects can be sustained. Stein highlights that “relations among institutional constructs, habits and the transformation of behavior are at the core of development.” of course, M & N also point out that changing behavior is not everything. behavior relates to what actors ‘will do’ but what they ‘can do’ is may still be constrained by a lack of material, human, and time resources.

M & N also draw on Bourdieu’s multiple forms of capital to consider the ways that interventions may change dispositions and behavior; these include economic but also cultural, social, and symbolic capital. the importance of non-economic forms of capital in changing motivations and incentivizing behavior are increasingly recognized, including the many talks of Rory Sutherland and Ashraf’s recent paper on pro-social benefits. the way an intervention will change the distribution of all these types of capital should be considered in the design and the process captured over the course of implementation.

the ways that previous interventions have altered the distribution of these forms of capital is also an important consideration; tabula rasa non existunt.

3. finally, for quite some time (with a little e-input from Owen Barder – thanks!), i have been working up to saying all social interventions – explicitly experimental or not – are experiments. they change the context in ways that are not adequately considered. this includes the ethical ramifications of implementing an intervention and of stopping that  intervention. at the very least, the presence of an experiment/intervention changes changes expectations. while i’ve been tiptoeing up to this point for months, M & N go right ahead and say, “short-term interventions can have long-term impacts on expectations.” an experiment or intervention might be billed as discrete, pilot, or otherwise, but if it is taking place outside a controlled laboratory setting, it is bringing about some change, from anchors and reference prices to much larger shifts in thinking on ‘how things work.’  experiments have social and political ramifications beyond the intended effects, at levels above the individuals directly involved.

to this end, M&N and others promote a ‘cyclical formative reformative research approach’ as a way of moving forwards with (health) systems research and i strongly back this idea as a way of experimenting and promoting development more generally. this sort of long-term research agenda does not always fit with the present structure of grants & funding but hopefully the latter will begin to change.