draft thoughts on showing the work over time in a theory of change (comments welcome)

in draft work with vegard iversen (see here), we have been developing some ideas around using (and showing) both ex ante and ex post theories of change. this is partly in line with a learning agenda for theories of change (as outlined in valters’ recent work here and in my follow-up here, among other places). a learning agenda includes both internal learning but also, and equally importantly in my view, a commitment to making learning public.

.

the overall argument we advance in the paper relates to bringing theory — both formal and programmatic — to the center of making claims about the external validity of evaluation findings. a small piece of this argument relates to reporting. i’d certainly be interested in views on the below, which is a slightly modified view of the text as it currently stands. i feel that much of the received wisdom is that higher and higher levels of abstraction are what make research findings portable across settings. and, yet, in many instances of discussing external validity (such as here), the conversation inevitably veers towards mixed methods.

.

Thicker (Geertz 1972) and richer descriptions about settings and implementation processes — linked with the ToC and its critical assumptions — will facilitate learning. The more deeply researchers can probe the local — and the more detail authors use describe it — the more the reader can try to assess the potential for generalization. Thickness, not thinness, helps users of evidence make this assessment in light of their own setting.

.

Indeed, Lincoln and Guba denote “thick description” as the main way of allowing results to be transferred, calling for “narrative developed about the [setting to allow] judgments about the degree of fit or similarity may be made by others who wish to apply all of part of the findings elsewhere” (Lincoln and Guba 1986).

.

We believe that similar concerns are what motivated Woolcock to close his paper on external validity with a call for more case study work (Woolcock 2013). Honesty about problems encountered and tweaks to implementation processes and quality made as a result of experiential learning and adaptation will facilitate lesson learning both ‘there’ and ‘here.’

.

The Theory of Change, a central premise of our arguments about taking external validity seriously, can provide a useful tool for structuring reporting and lesson learning. Said more plainly, we want to have ToCs play equally important roles at the beginning of a study and at the end. In his review of how Theories of Change are used, Valters writes: “as far as I know, there is no particular tool to go back and see if a ToC is right or not.” This lack of critical reflection clearly does not support lesson learning here nor there.

.

To this end, we put forward the idea that careful ex ante assembly and ex post refinement of a Theory of Change will assist in studies that rigorously increase their potential for external validity. Such reporting, perhaps particularly when structured by the ToC, would reflect the learning that happens between the design phase and during data collection and the analysis (Pritchett, Samji, and Hammer 2012).