reflecting on being radical: integrating theories of change as practice

last week, craig valters published new work on theories of change. he calls not for a new tool but for a more careful approach to practicing and engaging in development. that is, changing the state of the world for someone. and learning from it. and, ideally, communicating that learning. (craig is pessimistic that we are near actually ushering in a ‘learning agenda’ to replace the ‘results agenda.’ on this, i hope he is wrong.)


in this post, i aim to echo and expand on the idea of theories of change as allowing “space for critical reflection” (p. 4) and to push back slightly on two of the outlined ‘key principles’ of a theory of change approach: being ‘locally led’ and thinking ‘compass, not map.’


i should make two disclaimers, given points raised both in the paper and in suvojit’s follow-up blog. the first is a musing, though i have adopted the ‘theory of change’ language along with the herd. i wish we could still revise it to ‘hypotheses of change’ or ‘ideas of change’ or ‘stuff that might matter because we thought hard about it, looked at what had been done before, and talked to people about what could be done now.’ or something else catchier but far more tentative and humble than ‘theory.’ alas.


the second is a confession. i really like boxes and arrows. not as the definitive product associated with a theory of change but as some means of organizing ideas that people can stand around, look at, point to, and say, “have we learned anything about how this arrow really works?” while i wouldn’t want to foist the need for a visual on anyone, especially if it is just going to end as a bad flowchart, i feel i should at least lightly advocate that a visual can be a useful tool for learning and may be more friendly to revisit than a lengthy narrative. and be pretty.


spaces for critical reflection

i strongly agree with craig’s focus on process and learning, emphasizing the theory of change as an approach, not just a product. it requires a change in practice, not deliverables. lukewarm commitment will relegate theory of change products to templates and checklist items to complete: as valters warns against, suvojit worries about, and duncan green and others have questioned in the past. instead, the approach requires a physical and mental and temporal space actively held open to think and reflect and revisit.


to possibly only reword what has already been said: theories of change products will only matter for learning if the approach requires them to be as important at the beginning of the project as they are at the end. ‘doing’ theories of change requires thinking about the end at the beginning — but also to look backwards later.


theory of change products (narratives, diagrams) can and should be used to help organize learning along the way — whether revisited as part of the program diaries (great idea!) or as other structured (partly by the product), purposive ways of stopping the good work and reflecting on how we initially thought things would be working and how they are actually progressing. for me, with a goal of generating lessons from which others can learn, documentation is important. some may see this as a hindrance to engagement, so that remains an open question for each organization. in any case, i like craig’s questions related to ‘prioritizing learning,’ which encourage organizations (and donors) to be both explicit and transparent about learning goals (what, for whom, to make what decisions) from the outset.


one idea is to think about, as products, an ex ante theory of change and an ex post version. (the process of) producing these would, done well, encourage learning around where the two versions differ and would serve as a way to help organize learning about those differences. by “done well,” i emphasize my own principle of a theory of change approach, which is that a theory of change is never one person’s all-nighter, assembling something to submit as part of a report. to the extent that donors may play a role in encouraging grantees to reflect on how they thought change would happen and how it actually did (if, indeed, it did), they should. this means not just telling grantees to do this but facilitate in terms of finances and time and and convening power and guidance, as needed. active reflection is needed; sometimes external pressure can bring this about.


being locally led

by pushing back on being “locally led,” i am obviously not going to suggest that local voices and opinions shouldn’t be central to the process of figuring out what kind of change needs to brought about and how to go about it. but i would like to bring in some nuance as well generally advocate being led by multiple perspectives — top-down and bottom-up — rather than a singular idea of ‘the local.’ craig suggests some mild discomfort with this as well in his footnote that warns us against assuming there is an archetypal and wise end-user with all the answers.


first, local-ness is diverse (which almost goes without saying). iteratively, this diversity needs to be reflected in a theory of change process — and theories of change need to reflect this diversity. (with an evaluation hat on, what i am partly saying is that sub-groups in which heterogeneous treatment effects are expected should be visible early in the theory of change process.) beyond the tired statement that ‘the local’ — as possible recipients of a program/policy — is not a homogeneous mass, a key distinction is between ‘intended beneficiaries’ and ‘front-line implementers.’ both are local in important ways and the views and motivations of both should be reflected in theory of change processes and products (as noted by pritchett et al in considering structured experiential learning.)


second, local thinking can be constrained precisely by local-ness. researchers and practitioners are important conduits for transferring in good ideas from ‘there’ to consider whether they may work ‘here.’  being ‘locally rooted’ or ‘locally grounded’ may be slightly closer to my sense of an ideal than being ‘locally led,’ in part because…


third, i’d like to think there is still a role for theory (or Theory) in program theory and theories of change. again, practitioners and researchers play a role in helping to negotiate an intelligible space between (1) these ideas, (2) what cannot be done in a given local context (because of lacking physical, human, financial resources), and (3) what people don’t know if can be done locally because it has not been tried here in a particular way. ‘the local’ can best describe the various ways in which things have been happening and are critical voices in determining what should happen and generating ideas about what could happen — but it is not the only source of ideas and inspiration and energy and convening power.


thinking compass, not map

i understand this impulse for this principle — a reaction to over-design and over-planning — but i am going to push back a little anyway. or at least advocate for a compass plus a really amazing postcard* from the destination. my feeling is that organizations spend a lot of time talking about what they are going to do but not in visualizing or describing what the changed state of the world will look like and how people will behave differently in it if change actually happens. this (considering individuals, here and here) should be specific exercise within the approach, asking for different stakeholders: what do i do now when i wake up and go about my day and how will i do things differently if the intended good change comes about?


the discussion about how things will be different — and how we’ll know they are different — should be central to conversations about how change may happen. this can help to uncover some of the assumptions that are so important for theories of change. in addition, a detailed description of what change or success will look like is fairly important for understanding whether we’ve gotten to where we want to go (and if not, why not).


knowing where you’re going is different from saying you’re certain about how you will get there and when. over-planning and over-design reflect, i think, (donor-forced?) over-confidence about the route and mode and timing of transport. this is distinct from over-clarity about where you are trying to go, which is too often lacking. by all means, “go west;” carry with you some way of telling whether you’ve made it to oregon or not.


in sum

to briefly conclude, a big thank you to craig, who has done a great job laying out some of the promises and concerns of embracing theories of change as an approach to practicing and doing change. i echo the call to emphasize learning (with a distinction between failure to learn and the failure of a project to deliver) and add a few ideas about how the theory of change (as a product) might structure the learning process and its outputs. i explicitly call for the product to play a renewed role after the project launch, as a way of encouraging the process. i push back a bit on being ‘locally led,’ towards being ‘locally grounded’ and/but ‘multi-perspectival.’ i also push back on ‘compass, not map’ in favor of ‘compass and postcard,’ with explicit intent to encourage practitioners and researchers to have a good idea where they are going and to draw and annotate a (not the) map along the way.

*for jim gaffagan fans, this would be a postcard that goes beyond saying “this city has big buildings, i like food, bye” (around minute 2:40).

a small point on thinking about IEs by sector

in looking at how the accumulated impact evaluation evidence in the social sciences is distributed — perhaps with an eye toward making the case for where to concentrate new funding — there is a tendency to categorize studies by sector. with this lens, it is clear that the evidence base remains dominated by health, social protection, agriculture, sanitation.


this does not reflect the actual research questions or topics asked by these studies. for example, there are many ‘health’ studies — distributing health products, generally — but very few on health systems. the line is, of course, not clear: would vouchers to encourage entering the health system in terms of institutional/hospital-based delivery of babies fall in the distributing ‘stuff’ or building ‘systems’ category? such parsing would take further and careful thought.


nevertheless, for those looking at where the bulk of the literature falls — and where new research is needed — it may be helpful to move beyond sector codes (which are easier to find and seem to dominate the classification strategies of IEs) and start teasing apart ‘stuff’ from ‘systems’/institutions work.

Brief Thought on Commitment-To-Analysis Plans

First, I am starting a small campaign to push towards calling ‘pre-analysis plans’ something else before the train gets too far from the station. Something like ‘commitment to analysis plans’ or ‘commitment to analysis and reporting plans.’ I have two reasons for this.

  1. PAP just isn’t a super acronym; it’s kind of already taken.
  2. I think the name changes moves the concept a step back from indicating that the researcher needs to pre-specify the entirety of the analysis plan but, rather, to indicate the core intended dating cleaning and coding procedures and the central analysis — and to commit to completing and reporting those results, whether significant or not. this shift, from a commitment rather than a straitjacket, seems like it would go some way towards addressing concerns expressed by Olken an others that the task of pre-specifying all possible analyses ex ante is both herculean and blinkered, in the sense of not incorporating learning’s from the field to guide parts of the analysis. the commitment, it seems to me, should be partly around making clear to the reader of a study which analyses were ‘on plan’ and which came later, rather than claiming perfect foresight.

Second, speaking of those learning’s from the field that may be incorporated into analysis… I had a moment today to think a bit about the possible views from the field that come from surveyors (as I am working on doing some of my dissertation analysis and already starting to form a list of questions to write back to the survey team with which I worked!). Among the decisions laid out by folks like Humphreys and Mckenzie in their lists of what should be specified in a commitment to analysis plan (doesn’t a ‘CAP’ sound nice?) about data cleaning, surveyors play very little role.

Yet a survey (or discussion) among survey staff about their experience with the questionnaire can yield information on whether there were any questions with which they systematically felt uncomfortable or uncertain about or that respondents rarely seemed to understand. Yes, many of these kinks should be worked out during piloting but, no, they aren’t always. Sometimes surveyors don’t get up the gumption to tell you a question is terrible until the research is underway and sometimes they themselves don’t realize it.

For example, in one field experiment with which i was involved, surveyors only admitted at the end (we conducted an end-of-survey among them) how uncomfortable they were with a short-term memory test module (which involved asking respondents to repeat strings of numbers) and that it was quite embarrassing to ask these questions of their elders. To the point that some of them breezed through these questions pretty quickly during interviews and considered some of the answers they reported suspect. Some wrote fairly agonizing short essays to me in the end-of-survey questionnaire (it’s a good thing to make them anonymous!), asking me to “Imagine that you have to ask this question to an elder…” and proceeded to explain the extreme horror of this.* As the short-term memory module was not part of the central research question or main outcomes of interest, it was not subjected to any of the audit, back-check, or other standard data-quality procedures in place, and so the problem was not caught earlier.

I can imagine a commitment-to-analysis plan that committed to collecting and incorporating surveyor feedback. For example, a CAP that stated that if >90% of surveyors reported being uncertain about the data generated by a specific question, those data would be discarded or treated with extreme caution (and that caution passed on to the consumers of the research). Maybe this could be one important step to valuing, in some systematic way, the experience and insights of a survey team.

*For the record, I can somewhat imagine this, having used to work in a call center to conduct interviews with older women following up on their pelvic floor disorder surgery and whether they were experiencing any urinary symptoms. In that case, however, most of the discomfort was on my side, as they were well versed in — and fairly keen to — talking about their health issues and experiences! Note to self: aim not to have pelvic floor disorder.

Thinking More About Using Personas/Personae In Developing Theories of Change

I have previously advocated, here (and here), for taking a ‘persona’ or character-based approach to fleshing out a theory of change. This is a way of involving a variety of stakeholders (especially those closer to the ground, such as intended beneficiaries and street-level implementer’s) in discussions about program and theory of change development — even when they are not physically at the table, which is not always possible (though encouraged, of course).

This week, I had a new chance to put some of these ideas into action. A few lessons learned for future efforts:

  • This activity worked well in small groups. However, it may be too much to ask groups to fully develop their own personae, especially given possible time limits within the confines of a workshop.
    • It may be better to have some partially developed characters in mind (for example, that represent differing initial levels of the key outcomes of interest and variation on some of the hypothesized sub-groups of interest (explanatory variables). Groups can then take a shorter amount of time to elaborate — rather than develop — these dossiers and give a name to each of their creations (Mary, Bob, Fatima, etc). Alternatively, developing dossiers (and therefore articulating sub-groups of interest) could be a separate, opening activity.
  • Introducing any language about “role-playing” can lead to only one person in a group assuming the role of a given character and the group sort of playing ’20 Questions’ to that character, rather than everyone trying to consider and take on the thoughts, intentions, and decisions and steps a given character might take, confronted with a given intervention (as either a targeted beneficiary or an implementer). The idea is to get the team thinking about the potential barriers and enablers at multiple levels of influence (i.e. assumptions) that may be encountered on the path towards the outcomes of interest.
  • Speaking in “I” statements is helpful in helping people try to think like the different adopted personae. I really had to nag people on this in the beginning but I think it was ultimately useful to get people speaking in this way. In relation to this, there may be important lessons from cognitive interviewing (how-to here) practice, to get activity participants to think out loud about the chain of small decisions and actions they would need to take when confronted with a new program or policy.
  • I noted a marked tendency this time around for men to only speak for male characters and for women, the same! There may be some creative ways to discourage this (thoughts welcome).
  • There are two potential key goals of an activity like this, which should be kept distinct (and be articulated early and often during the activity) even though they are iterative.
    • A first relates to Elaborating Activities, that is, to develop a robust intervention, so that nuance to activities and ‘wrap-around’ support structures (to use Cartwright and Hardie’s terminology) and activities can be developed. This can lead to a laundry or wish list of activities — so if is at the brainstorming stage, this can be articulated as an ‘ok’ outcome or even an explicit goal.
    • A second relates to Explicating and Elaborating assumptions, filling in all the intermediate steps between the big milestones in a results chain. This second goal is bound up in the process of moving from a log-frame to a robust theory of change (as noted by John Mayne at the Canadian Evaluation Society, this is adding all the arrows to the results chain boxes) as well as a more robust and nuanced set of indicators to measure progress towards goals and uncover mechanisms leading to change.
      • A nice wrap-up activity here could include sorting out the assumptions for which evidence is already available and which should be collected and measured as part of research work.
  • It remains an important activity to elaborate and verbally illustrate how X character’s routines and surroundings will be different if the end-goals are reached — given that social, environmental, infrastructural and institutional change is often the goal of ‘development’ efforts. This last step of actually describing how settings and institutions may operate differently, and the implications on quotidian life, is an important wrap-up and time needs to be made for it.

Of course, the use of personae (or an agent-based perspective) is only one part of elaborating a theory of change. But it can play an important role in guiding the other efforts to provide nuance and evidence, including highlighting where to fill in ideas from theoretical and empirical work to end up with a robust theory of change that can guide the development of research methods and instruments.

Would be great to hear further ideas and inputs!

Back to Basics — Trusting Whether and How The Data are Collected and Coded

This is a tangential response to the lacour and #lacourgate hubbub (with hats off to the summaries and views given here and here). While he is not implicated in all of the comments, below, I am mostly certainly indebted to Mike Frick for planting the seed of some of the ideas presented below, particularly on member-checking (hopefully our under-review paper on the same will be out sometime in the future…). Salifu Amidu and Abubakari Bukari are similarly motivational-but-not-implicated, as are Corrina Moucheraud, Shagun Sabarwal and Urmy Shukla.

To a large extent, the lacour response is bringing a new angle on an increasingly familiar concern: trusting the analysis. This means additional (and important) calls for replication and other forms of post-publication peer review (as Broockman calls for) as a guard against significance-hungry, nefarious researchers. Pre-analysis plans, analytic/internal replications, and so on, are all important steps towards research transparency. But they miss the fundamental tendency to treat data as ‘true’ once it makes it into the familiar, rectangular format of a spreadsheet.

Given lacour, it seems clear that we may need to take an additional step back to get into the heart of research: the data. We place a lot of trust in data themselves — between advisers and advisees, between research collaborators, and between producers and users of large, public data sets. and, in turn, between PIs and research assistants and the actual team collecting the data. This trust is about, of course, whether the data exist at all and whether they measure what they purport to measure. (Green seems to have had a hunch about this?)

We should be clear about the foundations of this trust and what we might do to strengthen it. Ultimately, the lacour story is a story about the production of data, not its analysis. The transparency agenda needs to expand accordingly, to address the fundamental constancy that ‘shit in leads to shit out.’

Here’s a few thoughts:

  • Start to teach data collection like it matters. Survey design and data collection are weirdly absent from many graduate programs — even those oriented towards research. You may pick these up in electives but they are rarely required, to my knowledge. Learning about construct validity, validating test instruments in new contexts, questionnaire design, the potential for interview effects, some of the murky and inchoate contents of the activity labelled as ‘formative work*,’ etc, need not be re-discovered by each new graduate student or research assistant who takes on field work. If a course-work model won’t work, then a much more explicit apprenticeship model should be sought for those pursuing primary empirical work. in terms of teaching, one occasionally might be forgiven for thinking that impact evaluators had discovered data collection and that there aren’t mounds of resources on household surveys, psychometric’s, and questionnaire design that can be used to better ensure the quality and truthfulness of the data being collected. Interdisciplinary work needs to start with what and by what means and measures data are collected to answer a particular question.
  • Report on data quality practices. Lots of survey firms and researchers employee strategies such as data audits and back-checks. Good on you. Report it. This almost never makes it into actual publications but these are not just internal operations processes. Researchers do need to put forth some effort to make their readers trust their data as well as their analysis but so much less work seems to go into this. With the rise of empirical work in economics in other fields, this needs to be given more documented attention. If you threw out 5% of your data because of failed back-checks, tell me about it. I’d believe the remaining 95% of your data a lot more. The onus is on the researchers to make the reader trust their data.
  • Treat surveyors as a valuable source of information. It is increasingly common to at least have surveyors fill a question at the end of questionnaire about whether the respondent was cooperative (usually a Likert scale item) or other brief reflection on how the interview went. I have no idea what happens to responses to the data so produced — if they are used to throw out or deferentially weight responses, do please tell the reader about it. Moreover, you can systematically ask your surveyors questions (including anonymously) about question items that they don’t trust. For example, I asked (in written form) this question of surveyors and most reported that it was incredibly embarrassing for them to ask their elders to play certain memory games related to short-term recall. This might be a good sign to tread lightly with those data, if not discount them completely (whether or not the surveyors faithfully asked the embarrassing question, it still suggests that it created a tense social interaction that may not have generated trustworthy data, even if it didn’t fall in the traditional space of ‘sensitive questions.’). If nothing else, the surveyors’ assessments may be given as part of the ‘methods’ or ‘results’ attended to in publications. And, in general, remembering that surveys are human interactions, not matrix populators, is important.
  • Member-check. Member-checking is a process described by Lincoln and Guba (and others) that involves taking results and interpretations back to those surveyed to test interpretative hypotheses, etc. if some results really fly in the face of expectations, this process could generate some ‘red flags’ about which results and interpretations should be treated with care. And these can be reported to readers.
  • Coding. As with ‘formative work,’ the nuances of ‘we coded the open-ended data’ is often opaque, though this is where a lot of the interpretive magic happens. This is an important reason for the internal replication agenda to start with the raw data. In plenty of fields, it would be standard practice to use two independent coders and to report on inter-rater reliability. This does not seem to be standard practice in much of impact evaluation. This should change.
  • Check against other data-sets. It would not take much time for researchers to put into context their own findings by comparing (as part of a publication) the distribution of results on key questions to the distribution from large data-sets (especially when some questionnaire items are designed to mimic the dhs, lsms, or other large public data-sets for precisely this reason). This is not reported often enough. This does not mean that the large, population-linked data-set will always trump your project-linked data-set but it seems only fair to alert your readers to key differences, for the purposes of internal believability as well as external validity.
  • Compare findings with findings from studies on similar topics (in similar contexts) — across disciplines. Topics and findings do not end with the boundaries of a particular method of inquiry. Placing the unexpectedness of your findings within this wider array of literature would help.
  • Treat all types of data with similar rigor and respect. (Cue broken record.) If researchers are going to take such care with quantitative data and then stick in a random quote as anec-data in the analysis without giving any sense of where it came from or whether it should be taken as representative of the entire sample or some sub-group… well, it’s just a bit odd. However you want to label these different types of data — quant and qual or data-set-observations and causal-process observations — they are empirical data and should be treated with the highest standards known in each field of inquiry.

I can’t assess whether any of these measures, singly or together, would have made a major difference in the lacour case — especially since it remains nebulous how the data were generated, let alone with what care. But the lacour case reveals that we need to be more careful. A big-name researcher was willing to trust that the data themselves were real and collected to the best of another researcher’s ability — and focused on getting the analysis right. In turn, other researchers bought into both the analysis and the underlying data because of the big-name researcher. This suggests we need to do a bit more to establish trust in the data themselves — and that the onus for this is on the researchers — big names or no — claiming to have led the data collection and cleaning processes. This is especially true given the unclear role for young researchers as potential replicators and debunkers, highlighted here. I hope the transparency agenda steps up accordingly.

*If on occasion a researcher reported on what happened during the ‘formative phase’ and about how the ‘questionnaire was changed in response,’ that would be really interesting learning for all of us. Report it. Also, if you are planning to do ‘qualitative formative work’ to improve your questionnaire, it would be good if you built in time in your research timeline to actually analyze the data produced by that work, report on that analysis, and explain how the analysis led to changing certain questionnaire items…

Thinking About Building Evaluation Ownership, Theories of Change — Back From Canadian Evaluation Society

This week I had the pleasure of attending the Canadian Evaluation Society (#EvalC2015) meeting in Montreal, which brought together a genuinely nice group of people thinking not just hard a-boot evaluation strategies and methodologies but also how evaluation can contribute to better and more transparent governance, improving our experience as global and national citizens — that is, evaluation as a political (and even social justice) as much as a technical act.

Similarly, there was some good conversation around the balance between the evaluation function being about accountability versus learning and improvement and concern about when the pendulum swings too far to an auditing rather than an elucidating and improving role.

For now, I want to zoom in on a two important themes and more own nascent reflections on them. I’ll be delighted to get feedback on these thoughts, as I am continuing to firm them up myself. my thoughts are in italics, below.

  1. Collaboration, neutrality and transparency
    1. There were several important calls relating to transparency, including a commitment to making evaluation results public (and taking steps to make sure citizens see these results (without influencing their interpretation of them or otherwise playing an advocacy role)) and for decision-makers claiming to have made use of evidence to inform their decisions to be more open about how and which evidence played this role. This is quite an important point and it echoes some of the points Suvojit and I made about thinking about the use of evaluative evidence ex ante. We’re continuing to write about this, so stay tuned.
    2. There was quite a bit of push back about whether evaluation should be ‘neutral’ or ‘arm’s length’ from the program — apparently this is the current standard practice in Canada (with government evaluations). This push back seems to echo several conversations in impact evaluation about beginning stakeholder engagement and collaboration far earlier in the evaluation process, including Howard White’s consideration of evaluative independence.
    3. Part of the push back on ‘arm’s length neutrality’ came from J. Bradley Cousins, who will have a paper and ‘stimulus document’ coming out in the near future on collaborative evaluation that seems likely to be quite interesting. In another session, it was noted that ‘collaboration has more integrity than arm’s length approaches. I particularly liked the idea of thinking about how engagement between researchers and program/implementation folks could improve a culture of evaluative thinking and organizational learning — a type of ‘capacity building’ we don’t talk about all that often. Overall, I am on board with the idea of collaborative evaluation, with the major caveat that evaluators need to report honestly about the role the play vis-a-vis refining program theory, refining the program contents, assisting with implementing the program, monitoring, etc.
  2. Building a theory of change and fostering ownership in an evaluation.
    1. There was a nice amount of discussion around making sure that program staff, implementers, and a variety of stakeholders could “see themselves” in the theory of change and logic model/results chain. This not only included that they could locate their roles but also that these planning and communication tools reflected the language with which they were used to talking about their work. Ideally, program staff can also understanding their roles and contributions in light of their spheres of direct and indirect influence.
    2. John Mayne and Steve Montague made some very interesting points about building a theory of change, which I will have to continue to process over the upcoming weeks. they include:
      1. Making sure to think about ‘who’ in addition to ‘what’ and ‘why’ — this includes, I believe, who is doing what (different types and levels of implementer’s) as well as defining intended reach, recognizing that some sub-groups may require different strategies and assumptions in order for an intervention to reach them.
      2. As was noted “frameworks that don’t consider reach conspire against equity and fairness” because “risks live on the margin.” I haven’t fully wrapped my head around the idea of ‘theories of reach’ embedded or nested within the theory of change but am absolutely on-board with considering distributional expectations and challenges from the beginning and articulating assumptions about when and why we might expect heterogeneous treatment effects — and deploying quantitative and qualitative measurement strategies accordingly.
    3. John Mayne advocated his early thoughts that for each assumption in a theory of change, the builders should articulate a justification for its:
      1. Necessity — why is this assumption needed?
      2. Realization — why is this assumption likely to be realized in this context?
      3. This sounds like an interesting way to plan exercises towards collaborative building of theories of change
    4. a productive discussion developed (fostered by John Mayne, Steve Montague and Kaireen Chaytor, among others) around how to get program staff involved in articulating the theory of change. A few key points were recurring — with strong implications for how long a lead time is needed to set up an evaluation properly (which will have longer-term benefits even if it seems to be slightly inefficient upfront):
      1. Making a theory of change and its assumptions explicit is part of a reflective practice of operations and implementation.
      2. Don’t try to start tabula rasa in articulating the theory of change (‘the arrows’) with the implementing and program staff. Start with the program documents, including their articulation of the logic model or results chain (the ‘boxes’ in a diagrammatic theory of change) and use this draft as the starting point for dialogue.
      3. It may help to start with one-on-ones with some key program informants, trying to unpack what lies in the arrows connecting the results boxes. This means digging into the ‘nitty girtty’ micro-steps and assumptions, avoiding magical leaps and miraculous interventions. Starting with one-on-ones, rather than gathering the whole group to consider the results chain, can help to manage some conflict and confusion and build a reasonable starting point — despite the fact that:
      4. Several commentators pointed out that it is unimportant whether the initial results chain was validated or correct — or was even set up as a straw-person. Rather, what is important was having something common and tangible that could serve as a touchstone or boundary object in bringing together the evaluators and implementer’s around a tough conversation. In fact, having some flaws in the initial evaluators’ depiction of the results chain and theory of change allows opportunities for program staff to be the experts and correct these misunderstandings, helping to ensure that program staff are not usurped in the evaluation design process.
      5. Initial disagreement around the assumptions (all the stuff behind the arrows) in the theory of change can be productive if they are allowed to lead to dialogue and consensus-building. Keep in mind that the theory of change can be a collaborative force. As Steve Montague noted, “building a theory of change is a team sport,” and needs to be an iterative process between multiple stakeholders all on a ‘collective learning journey.’
        1. One speaker suggested setting up a working group within the implementing agency to work on building the theory of change and, moreover, to make sure that everyone internally understands the program in the same way.
    5. This early engagement work is the time to get construct validity right.
    6. The data collection tools developed must, must, must align with the theory of change developed collectively. This is also a point Shagun and i made in our own presentation at the conference, where we discussed our working paper on meaningfully mixing methods in impact evaluation. stay tuned!
    7. The onus is on the evaluator to make sure that the theory of change is relevant to many stakeholders and that the language used is familiar to them.
    8. There was also a nice discussion about making sure to get leadership buy-in and cooperation early in the process on what the results reporting will look like. Ideally the reporting will also reflect the theory of change.

Overall, much to think about and points that I will definitely be coming back to in later work. Thanks again for a great conference.

Oops, Got Long-Winded About ‘Median Impact Narratives’

*A revised version of this post is also available here.

I finally got around to reading a post that had been flagged to me awhile ago, written by Bruce Wydick. While I don’t think the general idea of taking sampling and representatives seriously is a new one, the spin of a ‘median narrative’ may be quite helpful in making qualitative and mixed work more mainstream and rigorous in (impact) evaluation.

Anyway, I got a bit long-winded in my comment on the devimpact blog site, so I am sticking it below as well, with some slight additions:

First, great that both Bruce and Bill (in the comments) have pointed out (again) that narrative has a useful value in (impact) evaluation. This is true not just for a sales hook or for helping the audience understand a concept — but because it is critical to getting beyond ‘did it work?’ to ‘why/not?’

I feel Bill’s point (“telling stories doesn’t have to be antithetical to good evaluation“) should be sharper — it’s not just that narrative is not antithetical to good evaluation but, rather, it is constitutive of good evaluation and any learning and evidence-informed decision-making agenda. And Bill’s right, part of the problem is convincing a reader that it is a median story that’s being told when an individual is used as a case study — especially when we’ve been fed outlier success stories for so long. This is why it is important to take sampling seriously for qualitative work and to report on the care that went into it. I take this to be one of Bruce’s key points and why his post is important.

I’d also like to push the idea of a median impact narrative a bit further. The basic underlying point, so far as I understand it, is a solid and important one: sampling strategy matters to qualitative work and for understanding and explaining what a range of people experienced as the result of some shock or intervention. It is not a new point but the re-branding has some important sex appeal for quantitative social scientists.

One consideration for sampling is that the same observable’s (independent vars) that drive sub-group analyses can also be used to help determine a qualitative sub-sample (capturing medians, outliers in both directions, etc). To the extent that theory drives what sub-groups are examined via any kind of data collection method, all the better. Authur Kleinman once pointed out that theory is what helps separate ethnography from journalism — an idea worth keeping in mind.

A second consideration is in the spirit of Lieberman’s call for nested analyses (or other forms of linked and sequential qual-quant work), using quantitative outcomes for the dependent variable to drive case selection, iterated down to the micro-level. The results of quantitative work can be used to inform sampling of later qualitative work, targeting those representing the range of outcomes values (on/off ‘the line’).

Both these considerations should be fit into a framework that recognizes that qualitative work has its own versions of representativeness (credibility) as well as power (saturation) (which I ramble about here).

Finally, in all of this talk about appropriate sampling for credible qualitative work, we need to also be talking about credible analysis and definitely moving beyond cherry-picked quotes as the grand offering from qualitative work. Qualitative researchers in many fields have done a lot of good work on synthesizing across stories. This needs to be reflected in ‘rigorous’ evaluation practice. Qualitative work is not just for pop-out boxes (I go so far as to pitch the idea of a qualitative pre-analysis plan).

Thanks to both Bruce and Bill for bringing attention to an important topic in improving evaluation practice as a whole — both for programmatic learning and for understanding theoretical mechanisms (as Levy-Paluck points out in her paper). I hope this is a discussion that keeps getting better and more focused on rigor and learning as a whole in evaluation, rather than quant v qual.