teaching qualitative analysis is not easy for several reasons. first, an awful lot of material on doing qualitative research focuses on data collection. relatedly, then, a lot of academic papers that draw on qualitative data and analytic methods focus on data collection and organization. too often the use of an analytic software stands in for an explanation of how analysis was done.

.

second, lecturing on qualitative analysis is much like a powerpoint lecture on riding a bicycle. it sounds very easy (right foot down, then left foot down). it only gets hard when you try to do it.

.

nevertheless, a lecture must begin somewhere. i hope my notes, below, may prove useful to someone else.

.

despite my impulse to start with lincoln and guba’s paper, since this wasn’t an audience that spends all the their time reading academic papers or thinking about theory, i started with an example published qualitative piece. i found one that focused on a similar data source and level (interviews with high-level stakeholders as opposed to, say, a focus groups in a village or historical document review) as well as stated analytic approach (in this case, this paper by smit et al. was a good match).

.

the intuition was that — even though this was not an audience entirely used to reading research outputs — it would be helpful to get a handle on the type of research product toward which we wanted to build before getting lost in the nitty gritty of analysis. with a slightly different audience, i probably would have made the lincoln & guba piece mandatory to provide a touchstone for considering and critiquing the paper and then for storyboarding our own paper.

.

after reviewing a few key terms central to doing qualitative work (sources of qualitative data (talk, text, observations, images), positionality, inductive reasoning, deductive reasoning, codes and coding), we spent much of the first day discussing and critiquing the paper. 

.

first, individually and then in pairs, and then finally as a big group, we explored these questions:

• what are the goals set out by the researchers for this project and paper
• why did a qualitative approach make sense to answer these questions?
• what are the key conclusions the researchers draw from their analysis?
• what types of data do the researchers use to support their conclusions?
• what types of analyses do the researchers use to support their conclusions?
• what is convincing about the link between the researchers’ results and their conclusions? could anything have been done to make this more convincing?
• do the researchers achieve the goals they set out for themselves? why or why not? what could have been done differently?

.

then, as a larger group, we explored these additional questions (which are mostly notes to myself of topics to cover rather than and handout distributed to to participants; the first set of questions i did distribute):

• methods: data collection, organization, analysis
  • who were the data collectors? can we tell from the paper? how?
  • what was the positionality of the interviewers vis-à-vis the informants? what difference does this make?
  • what data collection strategy was used?
  • why do key informant interviews make sense as a data source given the research questions and goals?
  • were any other types of data used? how?
  • how were key informants chosen?
    • how many interviews were completed?
    • what does purposive sampling mean? snowball sampling?
    • how do the authors signal that the sample is representative of the relevant interests (i.e., what is thematic saturation or redundancy? what does this imply about the relationship between data collection, entry, and analysis in qualitative research?)? is this convincing? could it have been more convincing and if so, how?
  • how do the authors display their sample? is it helpful? what characteristics do they highlight and why? could it have been done better or differently?
  • what is a semi-structured interview guide? how does it differ from a completely structured or unstructured questionnaire or guide?
    • what types of questions did the researchers ask? how do we know? what else might we have liked to have known?
  • is there anything else we would have liked to have known about how the data were collected?
  • how were the data converted into transcripts? do the authors provide all the information we want on this?
  • what does it mean in this case that an inductive approach was used?
    • how did the researchers set about their induction? is this convincing?
      • what does it mean that key themes “emerged”?
    • what would have been different if the researchers had used a deductive approach? how would the analysis have changed? what would have been the trade-offs?
    • what did the authors actually do in analysis? do they provide us enough information to know?

.

• results
  • how do the authors reassure us that the information from different stakeholders is used and presented in a balanced way? could this have been done differently or better? if so, how?
  • figure 1 is the main display of the (descriptive) results.
    • did you look at it carefully when reading the paper or did you skip over it?
    • where did the figure come from? what do the bullet points in each box represent?
    • is this figure meant to be descriptive or analytic?
    • what is helpful about this display? what could have been done differently?
  • how are the results in this paper organized?
    • how does the presentation of results relate to the research questions?
    • how are quotations used to communicate the results? is this effective? convincing?
    • how were the quotes selected? are they meant to be representative or exceptional? how do you know?
    • were conflicting or diverging viewpoints represented? how do you know?
    • do you feel the researchers have drawn reasonable inferences from the data?
    • do the conclusions follow from the data?
    • do you feel that the researchers already had the conclusions in mind before they analyzed the data? does this affect the convincing-ness of the analysis?

.

• interpretations
  • what did the researchers do to make the present paper credible?
  • what did the researchers do to make the present paper balanced?
  • what did the researchers do to enhancing transparency?
  • is the paper ultimately convincing? why/not?

*this is a joint post with suvojit, here.

.

avoiding “we saw the evidence and we made a decision…”

“…and that decision was: given that the evidence didn’t confirm our priors or show a program to be a success, to try to downplay and hide the evidence.”

.

before we dig into that statement (based-on-a-true-story-involving-people-like-us), we start with a simpler, obvious one: many people are involved in evaluations. we use the word ‘involved’ rather broadly. our central focus for this post is people who may block the honest presentation of evaluation results.

.

in any given evaluation, there are several groups of organizations and people with stake in an evaluation of a program or policy. most obviously, there are researchers and implementers. there are also participants. and, for much of the global development ecosystem, there are funders of the program, who may be separate from the funders of the evaluation. both of these may work through sub-contractors and consultants, bringing yet others on board.

.

our contention is that not all of these actors are currently, explicitly acknowledged in the current transparency movement in social science evaluation, with implications for the later acceptance and use of the results. the current focus is often on a contract between researchers and evidence consumers as a sign that, in ben olken’s terms, researchers are not nefarious and power (statistically speaking)-hungry (2015). to achieve its objectives, the transparency movement requires more than committing to a core set of analyses ex ante (through pre-analysis or commitment to analysis plans) and study registration.

.

to make sure that research is conducted openly at all phases, transparency must include engaging all stakeholders — perhaps particularly those that can block the honest sharing of results. this is in line with, for example, EGAP’s third research principle on rights to review and publish results. we return to some ideas of how to encourage this at the end of the blog.

.

now, back to the opening statement, a subversion of the goal of evidence-informed decision-making. there are many interesting ways that stakeholders may try to dodge an honest sharing of results once they know what the results are. one is to claim that the public — whether in office or general public — will not be able to make sense of the results, so anything confusing, or, really, unexpected, needs to be pruned from the public report. instead, all the not-as-hoped results can be relegated to internal rather than public, learning.

.

decision-makers may indeed need brief synopses (written or otherwise) rather than being presented with a long report. different combinations and permutations of the evidence may be presented to different stakeholders using different modes of communication, in line with what is salient to them.

.

however, this is not a suitable excuse to fail make the full set of findings public. moreover, an assessment of what stakeholders can/not interpret that fails to account for how they say they want to receive evidence misses a key point of participation and partnership. it might reveal our (mis-)estimation of the policymaker’s intelligence and the complex policy challenges decision-makers encounter as part of their daily work.

.

we’ve talked elsewhere about committing to a decision process informed by evidence. in this post, we are after something even more simple: for key stakeholders to commit ex ante to making the results of a commissioned study public, irrespective of their respective priors regarding the intervention being studied. of course, the piece of research should be deemed as technically sound. assuming that it is, the goal is to encourage the honest sharing of results regardless of the direction of the results.

.

in theory, everyone party to a good ex ante evaluation (and ex post, though there may be slightly less stakeholder engagement; or the degree of engagement could vary depending on the emerging results from the study) is aware that the results for the effect of an intervention on an outcome of interest can be as hoped, opposite, null, or otherwise mixed and confusing. in practice, everyone has a prior, which may involve not just an educated hypothesis but an emotional commitment to a particular outcome.­

.

so what can help reduce the impulse and potential to cover-up unexpected results?

1. better explanation of research processes and norms. in some cases, key actors within commissioning agencies may be initially enthusiastic about the idea of evaluation without fully understanding what it — and a measurement and results focus more generally — really entails. here, one often makes the mistake of focusing on agency-capacity, rather than the capacity of individuals within these agencies. by capacity, we refer not only to technical know-how of evaluation methods but also familiarity with research processes and norms. disparity in capacity can lead to serious contradictions within the same agency on the way research findings are treated.

.

too often, though, efforts at “capacity-building” and other modes of education for individuals within agencies about evaluation focus on evaluation designs and analysis. this comes at the expense of explaining the research process, the variety of possible evaluation outcomes, and norms around transparent reporting of results. patrick dunleavy recently outlined the process of storyboarding research from the get-go to improve working in teams and helping to visualize the end-product. such a process may be useful for a broader array of stakeholders than the research team, so that the whole process (the whole magic of “analysis and writing up”) can be made transparent. this represents a potentially softer, friendlier and more feasible version than drafting the entire report in advance, as humphreys et al. attempted in their paper on fishing. it also may allow more of the process to be visible, rather than just the final reporting structure.

.

2. invest time in bringing all stakeholders to understand and agree with the research objectives and processes. several research studies (especially evaluations) have a committee of advisers to steer the process. these are critical stakeholders in addition to those that commission and carry out the research. ideally, all of those involved — including this committee of advisers — would reach a common understanding of the research objectives and methods to be followed. this would also include identifying policy messages from the study and engagement strategies.

.

however, common ground is sometimes elusive, as these wider groups do not arrive early on at a fruitful working arrangement or basic understanding of the research process. setting clearly understood objectives and a shared understanding of research processes may be time consuming but is invaluable when seen in the context of decision-making and transparency over research findings that may not match everyone’s priors.

.

3. formal commitment to results reporting across stakeholders. right now commitment to analyses and results reporting exist between researchers and the public or, really, other researchers. but researchers are not the only ones determining the content of results reporting — and thus reporting requires additional sets of (public? formal? registered?) commitments. these could, like pre-analysis plans or commitment-to-analysis plans, take the form of committing to a core set of analyses and reporting on these results. it may also take the form of MOUs that are less technical than ex ante analysis plans but still represent a commitment to reporting a certain set of results regardless of the direction of those results.

.

in any case, the goal is to move the commitment from being between researchers (and perhaps mostly intelligible to researchers) to also involving study commissioners, other stakeholders with the power to block the publication of findings, and the public (such as the public paying the taxes to fund the program).

.

4. early engagement with decision-makers. if decision-makers are a primary audience for the evaluation and if communicating to decision-makers is seen as a barrier to a complete, nuanced presentation of evaluation findings, then engaging with decision-makers early on may help. we recognise the time constraints of decision-makers and the importance of clarity in messaging. but the clarity of presentation and the complicatedness of the results need not be zero-sum.

.

one way to reduce this tension and to better communicate complex or complicated findings to decision-makers is to engage them in the evaluation from the very beginning, so that the potential for nuanced findings can be gradually introduced. if faced with a passive policy audience at the end of an evaluation, whose only role has been to turn up to listen to research findings in a workshop, the space for taking in complexity, nuance, and caveats in messaging will be limited. but assuming that evaluations findings need to assert only non-complex finding and straightforward recommendation is hugely problematic since we are talking about evaluations in social systems. as such, getting early buy-in and opening channels to gradually introduce results are important.

.

with these steps in place, chances are better that our based-on-a-true-story colleagues could have avoided the scenario that we referred to at the beginning of this post. an early commitment to the research processes and an agreement on the way forward would have helped prime key stakeholders to the possibility that research findings might be a mixed bag — which necessitates a nuanced dissemination strategy but not the burying of unfavorable results.

in draft work with vegard iversen (see here), we have been developing some ideas around using (and showing) both ex ante and ex post theories of change. this is partly in line with a learning agenda for theories of change (as outlined in valters’ recent work here and in my follow-up here, among other places). a learning agenda includes both internal learning but also, and equally importantly in my view, a commitment to making learning public.

.

the overall argument we advance in the paper relates to bringing theory — both formal and programmatic — to the center of making claims about the external validity of evaluation findings. a small piece of this argument relates to reporting. i’d certainly be interested in views on the below, which is a slightly modified view of the text as it currently stands. i feel that much of the received wisdom is that higher and higher levels of abstraction are what make research findings portable across settings. and, yet, in many instances of discussing external validity (such as here), the conversation inevitably veers towards mixed methods.

.

Thicker (Geertz 1972) and richer descriptions about settings and implementation processes — linked with the ToC and its critical assumptions — will facilitate learning. The more deeply researchers can probe the local — and the more detail authors use describe it — the more the reader can try to assess the potential for generalization. Thickness, not thinness, helps users of evidence make this assessment in light of their own setting.

.

Indeed, Lincoln and Guba denote “thick description” as the main way of allowing results to be transferred, calling for “narrative developed about the [setting to allow] judgments about the degree of fit or similarity may be made by others who wish to apply all of part of the findings elsewhere” (Lincoln and Guba 1986).

.

We believe that similar concerns are what motivated Woolcock to close his paper on external validity with a call for more case study work (Woolcock 2013). Honesty about problems encountered and tweaks to implementation processes and quality made as a result of experiential learning and adaptation will facilitate lesson learning both ‘there’ and ‘here.’

.

The Theory of Change, a central premise of our arguments about taking external validity seriously, can provide a useful tool for structuring reporting and lesson learning. Said more plainly, we want to have ToCs play equally important roles at the beginning of a study and at the end. In his review of how Theories of Change are used, Valters writes: “as far as I know, there is no particular tool to go back and see if a ToC is right or not.” This lack of critical reflection clearly does not support lesson learning here nor there.

.

To this end, we put forward the idea that careful ex ante assembly and ex post refinement of a Theory of Change will assist in studies that rigorously increase their potential for external validity. Such reporting, perhaps particularly when structured by the ToC, would reflect the learning that happens between the design phase and during data collection and the analysis (Pritchett, Samji, and Hammer 2012).