Thinking About Building Evaluation Ownership, Theories of Change — Back From Canadian Evaluation Society

This week I had the pleasure of attending the Canadian Evaluation Society (#EvalC2015) meeting in Montreal, which brought together a genuinely nice group of people thinking not just hard a-boot evaluation strategies and methodologies but also how evaluation can contribute to better and more transparent governance, improving our experience as global and national citizens — that is, evaluation as a political (and even social justice) as much as a technical act.

Similarly, there was some good conversation around the balance between the evaluation function being about accountability versus learning and improvement and concern about when the pendulum swings too far to an auditing rather than an elucidating and improving role.

For now, I want to zoom in on a two important themes and more own nascent reflections on them. I’ll be delighted to get feedback on these thoughts, as I am continuing to firm them up myself. my thoughts are in italics, below.

1. Collaboration, neutrality and transparency. 
   1. There were several important calls relating to transparency, including a commitment to making evaluation results public (and taking steps to make sure citizens see these results (without influencing their interpretation of them or otherwise playing an advocacy role)) and for decision-makers claiming to have made use of evidence to inform their decisions to be more open about how and which evidence played this role. This is quite an important point and it echoes some of the points Suvojit and I made about thinking about the use of evaluative evidence ex ante. We’re continuing to write about this, so stay tuned.
   2. There was quite a bit of push back about whether evaluation should be ‘neutral’ or ‘arm’s length’ from the program — apparently this is the current standard practice in Canada (with government evaluations). This push back seems to echo several conversations in impact evaluation about beginning stakeholder engagement and collaboration far earlier in the evaluation process, including Howard White’s consideration of evaluative independence.
   3. Part of the push back on ‘arm’s length neutrality’ came from J. Bradley Cousins, who will have a paper and ‘stimulus document’ coming out in the near future on collaborative evaluation that seems likely to be quite interesting. In another session, it was noted that ‘collaboration has more integrity than arm’s length approaches. I particularly liked the idea of thinking about how engagement between researchers and program/implementation folks could improve a culture of evaluative thinking and organizational learning — a type of ‘capacity building’ we don’t talk about all that often. Overall, I am on board with the idea of collaborative evaluation, with the major caveat that evaluators need to report honestly about the role the play vis-a-vis refining program theory, refining the program contents, assisting with implementing the program, monitoring, etc.
2. Building a theory of change and fostering ownership in an evaluation.
   1. There was a nice amount of discussion around making sure that program staff, implementers, and a variety of stakeholders could “see themselves” in the theory of change and logic model/results chain. This not only included that they could locate their roles but also that these planning and communication tools reflected the language with which they were used to talking about their work. Ideally, program staff can also understanding their roles and contributions in light of their spheres of direct and indirect influence.
   2. John Mayne and Steve Montague made some very interesting points about building a theory of change, which I will have to continue to process over the upcoming weeks. they include:
      1. Making sure to think about ‘who’ in addition to ‘what’ and ‘why’ — this includes, I believe, who is doing what (different types and levels of implementer’s) as well as defining intended reach, recognizing that some sub-groups may require different strategies and assumptions in order for an intervention to reach them.
      2. As was noted “frameworks that don’t consider reach conspire against equity and fairness” because “risks live on the margin.” I haven’t fully wrapped my head around the idea of ‘theories of reach’ embedded or nested within the theory of change but am absolutely on-board with considering distributional expectations and challenges from the beginning and articulating assumptions about when and why we might expect heterogeneous treatment effects — and deploying quantitative and qualitative measurement strategies accordingly.
   3. John Mayne advocated his early thoughts that for each assumption in a theory of change, the builders should articulate a justification for its:
      1. Necessity — why is this assumption needed?
      2. Realization — why is this assumption likely to be realized in this context?
      3. This sounds like an interesting way to plan exercises towards collaborative building of theories of change
   4. a productive discussion developed (fostered by John Mayne, Steve Montague and Kaireen Chaytor, among others) around how to get program staff involved in articulating the theory of change. A few key points were recurring — with strong implications for how long a lead time is needed to set up an evaluation properly (which will have longer-term benefits even if it seems to be slightly inefficient upfront):
      1. Making a theory of change and its assumptions explicit is part of a reflective practice of operations and implementation.
      2. Don’t try to start tabula rasa in articulating the theory of change (‘the arrows’) with the implementing and program staff. Start with the program documents, including their articulation of the logic model or results chain (the ‘boxes’ in a diagrammatic theory of change) and use this draft as the starting point for dialogue.
      3. It may help to start with one-on-ones with some key program informants, trying to unpack what lies in the arrows connecting the results boxes. This means digging into the ‘nitty girtty’ micro-steps and assumptions, avoiding magical leaps and miraculous interventions. Starting with one-on-ones, rather than gathering the whole group to consider the results chain, can help to manage some conflict and confusion and build a reasonable starting point — despite the fact that:
      4. Several commentators pointed out that it is unimportant whether the initial results chain was validated or correct — or was even set up as a straw-person. Rather, what is important was having something common and tangible that could serve as a touchstone or boundary object in bringing together the evaluators and implementer’s around a tough conversation. In fact, having some flaws in the initial evaluators’ depiction of the results chain and theory of change allows opportunities for program staff to be the experts and correct these misunderstandings, helping to ensure that program staff are not usurped in the evaluation design process.
      5. Initial disagreement around the assumptions (all the stuff behind the arrows) in the theory of change can be productive if they are allowed to lead to dialogue and consensus-building. Keep in mind that the theory of change can be a collaborative force. As Steve Montague noted, “building a theory of change is a team sport,” and needs to be an iterative process between multiple stakeholders all on a ‘collective learning journey.’
         1. One speaker suggested setting up a working group within the implementing agency to work on building the theory of change and, moreover, to make sure that everyone internally understands the program in the same way.
   5. This early engagement work is the time to get construct validity right.
   6. The data collection tools developed must, must, must align with the theory of change developed collectively. This is also a point Shagun and i made in our own presentation at the conference, where we discussed our working paper on meaningfully mixing methods in impact evaluation. stay tuned!
   7. The onus is on the evaluator to make sure that the theory of change is relevant to many stakeholders and that the language used is familiar to them.
   8. There was also a nice discussion about making sure to get leadership buy-in and cooperation early in the process on what the results reporting will look like. Ideally the reporting will also reflect the theory of change.

Overall, much to think about and points that I will definitely be coming back to in later work. Thanks again for a great conference.