learning by asking: a modest proposal to engage those who did the doing

this post was originally published at innovations for poverty action. it represents an effort to follow-through on some themes i have raised before and hope to continue to raise.

most development interventions are carried out and delivered by local research staff and residents. such implementation is rarely a straightforward ‘technical’ operation but, rather, there is social and political nuance in translating an idea into practice. on-the-ground partners therefore often have important insights about research engagement and implementation processes. but researchers often do not solicit such feedback as a requisite part of their work. moreover, even when such questions are asked, the results are often not synthesized into the lessons learnt from the project, either in terms of setting the future research agenda or in terms of tweaking on-going programs and interventions.

the Preserving ACTs (PACT) impact evaluation used text messages (via this code, built by @eokyere and @slawarokicki) to encourage the completion of the most effective anti-malarial treatment (artemisinin-based combination therapies, or ACTs (e.g.)) in and around Tamale, Ghana. as the study drew to a close, we wanted to thank the malaria-drug vendors for implementing the text-message intervention under investigation and for hosting the study’s surveyors. in addition, we wanted to solicit their feedback about the project and research process. our goals were to better understand their experience working with the research team and the study process, of the intervention process, and whether, how, and with what changes they thought the intervention should be scaled following the study.

photo credit: alidu osman  tuunteya

photo credit: alidu osman tuunteya

in mid- to late- 2011, one of our field managers went back to all 73 vendors – hospitals, clinics, pharmacists, and licensed chemical shops, both public and private – that had hosted our study, to thank them and to solicit their feedback on the study process and PACT intervention. 65 vendors were available and willing to be interviewed. these follow-ups took place in the third and fourth week of October 2011 and provide a unique set of data for considering the implementation of this and future studies. they also shed some light on the program that provided the context and basis for this operational research study – the Affordable Medicines Facility – malaria (AMFm) — discussed more at the end of this post.

as a token of our appreciation for their hosting PACT and our surveyors (and the only incentive provided to vendors for hosting the study and intervention), we provided a thank you note / certificate of participation. a few of the vendors noted that our follow-up was appreciated, such as “we thought you left without a goodbye” and  “I really never thought you would be back again after that long time.  But it is great you still remembered us and even come to officially say goodbye to us.”

as described in the paper, drug vendors played an important role in the study, which spanned much of the rainy season in northern Ghana as well as the Muslim fasting period of Ramadan, as mentioned in some of the follow-up interviews.

first, vendors were in charge of distributing the flyers to patients acquiring a malaria treatment. from this flyer, the patient would know to call or ‘flash’ (ring once for no charge) into our ‘mobile health information system,’ from which they were randomized to receive reminder text messages.

second, vendors provided an IPA surveyor a place to sit and talk to patients about their anti-malarial treatment and, if eligible, to recruit them into the study. Sometimes this meant giving them a chair in a hospital waiting room; other times, it meant the IPA team providing a chair for a surveyor to sit in the limited space offered by a one-man or one-woman drug shop.

given the study design, the distribution of flyers and the text messages were kept conceptually and, as feasible, spatially, separate from the IPA surveyor and his or her questions about the patient’s malaria medication. we included a question about this in the follow-up: the full range of vendor responses can be viewed here [link to the spreadsheet], as well as an overview of how responses were coded.* in short, the codes used to categorize responses emerged from the responses themselves in light of themes or ideas repeated across several answers.

given the sample-size needs of the study, surveyors were only placed at vendors that had ACTs in stock and were selling at a reasonable volume. this often meant longer stays at some of the hospitals, clinics, and popular pharmacies and LCSs but shorter stays with other vendors. some of the vendors expressed disappointment about this, such as, “I felt the surveyors did not spend much time in my shop.  You should have stationed a surveyor at my shop for at least three weeks or 4 weeks” and “I did not experience much due to a lack of patients during the study period at my shop.”

experiences with PACT.

though we did not explicitly ask vendors if PACT addressed a problem they thought was important, three-quarters volunteered that non-completion of ACTs was a problem they faced and many welcomed the support of a program like PACT to try to encourage people to finish their medication and to strengthen whatever advice they already gave to their patients – for example, saying that the text messages “made patients to take my advice more seriously” and that they “increase our credibility and patients to stick to these advice we give as a health service provider”.

others felt that the text messages led to patients feeling they were receiving addition concern for their wellbeing. one noted that PACT “gave patients the courage to come back to us and ask questions of the drugs given to them, and then some do come back later to report other issues” and another that “patients that did initially not want to participate have come later to commend the text reminder.”

prior to agreeing to host PACT, some vendors were concerned that the flyer and survey process would interrupt the patient flow, affect their business, or lead to an over-focus on malaria patients at the expense of other clients. after the study, about 40% felt their sales stayed normal during the study while about 20% felt their sales had increased, which they attributed to PACT. Nobody reported a disruption in their sales. about 40% did note that the questionnaires administered by surveyors were long and were uncomfortable for patients who were sick and tired – hence, being at a drug vendor in the first place.

about 20% also felt their status or credibility increased when they were offering the program, such as “business was normal but on social grounds, I earned respect from individuals that were sent SMS…  It was a remarkable experience of the concern they achieved from your end. So, I gained more credibility to my advantage.” some of these benefits may, of course, reflect a novelty effect and would not offer any one vendor an advantage if the program scaled-up to all vendors.

there were also a few unexpected benefits of the program likely not to be replicated in a scaled-up version of the program — for example, one vendor reported that “your surveyor also helped me track a thief that came to steal from me”!

several vendors highlighted that it’s not always easy for them when a study like this – which facilitates their providing an addition service to patients and clients — ends, noting, for example, that “some patients still ask me if they will get the SMS” and that “now I have to explain to my clients that I cannot send SMS to remind them now, which is sort of sad”.

continuing and scaling-up program.

most vendors (97%) explicitly said that it would be good the PACT program – or something like it – could be extended beyond Tamale and the time period of the study but also to other medications. for example, one noted “it should be scaled up so that at least I know my family that does not reside here in Tamale would also benefit from this great experience and education” while another said “the program should be nationwide because I know this attitude” – “the bad attitude of not completing the dosage” – “is widespread” and another similarly stated “the issue of people not completing their dosage is nationwide, so the program must be nationwide”.

about half saw no predictable barriers in being able to continue a program like PACT on their own; nearly a quarter thought that during busy times, it might be hard to make time to explain the program to patients and enroll them. 42% said they would need more training. Half noted that materials (such as flyers) and money for credit would need to be provided to them and several noted that extra components, such as t-shirts, would help motivate them and advertise the program better. for example, “I would also love to be given [something extra], like t-shirts, that could help motivate the vendors to do the work diligently.”

vendors also had a variety of suggestions on how enrollment into the text messaging system could be encouraged. One set of suggestions including removing any costs to patients, such as making the enrollment line toll-free as a way of encouraging more patients to enroll. to clarify what happened in practice, if patients texted into the program, their phone balance was deducted the cost of a text message. for people to ‘flash’ (dial and allow one ring) into the system, it did not cost them any money but phones (rather than our messaging system) required a positive balance to make this call.

more generally, 37 vendors (60%) suggested ways of increasing enrollment into the text messaging program, including the vendor or PACT team doing the actual enrollment as well as providing more information to patients about the program and why it was important. for example, one noted “if people are educated well, they will charge their phones and enroll later.” given the goal of the study — to see whether and how many people would enroll with a limited amount of encouragement and input — vendors provided limited (and surveyors provided no) information to the patient along with the flyer. in retrospect, however, some vendors saw this as a missed opportunity to educate patients about an important problem.

fourteen of the vendors raised concern that the text messages excluded illiterate patients – some suggested using voice calls as well as TV and radio to better educate about the importance of completing an ACT once it is started.

distinctions with AMFm (and implied suggestions for future AMFm-like programs).

the Affordable Medicines Facility – malaria (AMFm) provided the context for the PACT study, although the text messaging program was meant to be seen as independent of the ‘green leaf’ efforts explicit to AMFm marketing. as such, we asked questions of vendors to assess whether they saw the programs as distinct. In so doing, they ended up commenting on differences in the program in a way that may shed light on future AMFm-like efforts. for example, “programs for the ‘green leaf’ have more TV and radio coverage than your PACT program, but then you made more impact of encouraging people to take their drugs.  The other programs only encourage buying and awareness” or that with AMFm, “people do not get education on usage or encouragement to complete dosage”.

to clarify, the AMFm-linked advertisements focused on telling people to act fast when they suspected they had malaria, to look for the “green-leaf” quality logo, and what price these logoed ACTs should cost. as such, one vendor noted “the ‘green leaf’ programs don’t have any sort of attachment to the patient but your intervention seemed to involve and appreciate the patients more, so they felt respected and began to place some emphasis on their health.”

also, it is worth noting that elements of PACT seemed to alter the way patients viewed ACTs as well as the perceptions and stocking decisions of the vendors themselves. overall, 65% of vendors reported such changes. as part of AMFm, vendors should have received trainings provided by various government and professional bodies, as well as the mass media “green leaf” advertisements — but this seems to have not always been the case, at least by the time the PACT study was carried out.

for example, one vendor noted that “people are now more confident in the ACT drugs” and that “I now keep more stock of ACTs and on days that I do not have ACTs, the clients disturb me so much.  So for now, I make sure to keep more stock of ACTs.” Another said, “to be honest, it was the start of your study and the subsequent positioning of a surveyor at your shop that made me to purchase ACTs, and now I have many varieties of ‘green leaf’ drugs” while another reported “we were a bit skeptical about the new ACTs, so we had few in stock, but your presence also made us want to be more serious and also keep different types of ACTs.”

overall, the data presented above and in the spreadsheet (available below) suggest there is much that can be learned from soliciting feedback from program implementers, which can be taken into account in designing future studies and interventions. this type of engagement can ensure that humanity and humility remain part of the implementation and learning process.

you can download the spreadsheet with more information and responses here (excel format).

*where a vendor directly identified themselves or their workplace in their response, we have slightly altered the response, using italics in place of the missing word or phrase.

**important thanks go to Jeff Mosenkis, Corrina Moucheraud, Elianne Oei, John Quattrochi, and the late Elif Yavuz for encouragement in writing-up and thinking-through these results. the data and photographs above were collected by Usama Salifu and Alidu Osman Tuunteya. the biggest thanks go to the vendors for taking the time and energy to provide additional insight.

planning for qualitative data collection and analysis

this blog reflects conversations and on-going work with both mike frick (@mwfrick), shagun sabarwal (@shagunsabarwal), and urmy shukla (@urmy_shukla) — they should receive no blame if this blog is wacky and plenty of credit if it is not.

a recent post by monkey cage contributors on the washington post, then summarized by BITSS, asked/suggested whether “exploratory, qualitative, historical, and case-based research is much harder to present in a results-free manner, and perhaps impossible to pre-register.” this was just a brief point in their larger argument, which i agree with. but it seems worth pausing to consider whether it is likely to be true. below, i discuss research design for qualitative work, sampling considerations, and analysis itself.

throughout, i take a ‘pre-analysis plan’ to be a commitment on what the research will analyze and report on but not a constraint on doing analyses that are ‘off plan.’ rather, the researcher just needs to be explicit about what analyses were ‘on plan’ and which were not and there is a commitment to report everything that was ‘on plan’ – or why such reporting is infeasible.

my conclusion: a conversation on pre-analysis plans needs to distinguish whether planning is possible from whether planning is currently done. in general, I feel planning for analysis is possible when you plan to do analysis.

disclaimer: in this post, my reference to ‘qualitative research’ is to the slice of social science research that has to do with talking to and/or observing and/or participating with living people. i claim no knowledge on the analyses of historical manuscripts and a wide array of other qualitative research. by extension, i am mostly talking about planning for the collection and analysis of data from in-depth interviews, focus group discussion, and forms of (participant-) observation.

designing research: working with and asking questions of living people implies an ethics-review process, for which the research will have to lay out at least the ‘domains’ (aka themes, topics, categories) of information s/he hopes to observe and ask people about. usually, one does not get away with saying “i am just going to head over here, hang out, and see what i find.” this requires planning. like a pre-analysis plan, the domains for qualitative work can set up some bounds for the minimum of what will be collected and reported (“we will collect and analyze data on the following topics: x, y, z“), even if the final report is that a particular domain ended up being a flop because no one wanted to talk about it or it proved uninteresting for some reason.

some of the most famous ethnographies (say, nisa and tuhami) focus on a single person, often to try to give perspective on a larger culture — which may not be what the ethnographer initially set out to study. but the ethnographer can still tell you that (“i went to look at x, and here’s what i found — but i also found this really interesting person and that’s what the rest of the book is about”). so this does not seem inconsistent with the underlying logic of a plan, with the understanding that such a plan does not dictate everything that follows but does mandate that one reports why things changed.

which brings us to the nature of qualitative work: it is often iterative and the researcher often conducts data collection, entry, and analysis in parallel. analysis from an early set of interviews informs questions that are asked and observations attended to later on. this is one of the exciting (to me) elements of qualitative research, that you get to keep incorporating new learnings as you go along. this approach need not be inconsistent with having a set of domains that you intend to explore. within each, maybe the questions get sharper, deeper, or more elaborate over time. or maybe one planned domain turns out to be way too sensitive or way off-base. again, the researcher can report, relative to the initial plan, that this is what happened between designing the research and actually doing it.

sampling: certain aspects of qualitative research can be planned in advance. usually the aim is to be in some way representative. one way to aim for representation is to consider sub-groups of interest. in large-n analysis, the researcher may be able to hope that sufficient numbers of sub-groups will appear in the sample by default. in smaller-n analysis, more purposive sampling plans may be needed to be sure that different sub-groups are engaged in conversation. but, specifying sub-groups of interest can be done in advance — hence, plannable. but, at least some branches of qualitative research suggest that representativeness is about outputs rather than inputs — and that what the researcher is seeking is saturation (i am thinking of lincoln and guba here), which has implications for planning.

‘saturation’ relates to whether the researcher is starting to here the same answer over and over. in some cases, inputs are the determinant of representation — similar to the approach that can be taken in large-n work. let’s say that you want to get the input of the members of an elite government entity –  a particular committee with 20 people on it. fine, plan to talk to all of them. representativeness is here achieved by talking to all of relevant people (the whole population of interest) – and then finding someway of summarizing and analyzing the viewpoints of all of them, even if it’s 20 different viewpoints. there’s your sampling plan. (this may or may not be part of a pre-analysis plan, depending how that is defined and at what stage of the research process it is required. i take these to be open questions.)

for less clearly bounded groups that nevertheless have clear characteristics and may be expected to think or behave differently — let’s say men versus women, older versus younger people, different economic quantiles, different ethnic groups, whatever — then planning for saturation may look more like: plan to talk to men until we start getting repeated answers on key questions or interest or conduct focus groups that are internally homogenous with respect to ethnicity until we start to hear similar answers within each ethnicity (because it may be different numbers within each). that is, if representativeness is focused on output, then it is insufficient to plan at the beginning “we will do two focus groups in each village in which we collect data.” the researcher can specify the sub-groups of interest but probably not number of interviews, focus groups, or hours of observation required.

i make this point for two reasons. first, a pre-analysis plan for qualitative work should plan for iteration between considering what has been collected and whether more questions are necessary to make sense of the phenomena of interest. this makes it different in practice than a quantitative plan but the underlying principle holds. second, a pre-analysis plan, if it covers sampling, probably cannot plan for specific numbers of inputs unless the population is clearly bounded (like the committee members). rather, the plan is to aim for saturation within each sub-group of interest.

analysis: finally! in general, i feel more emphasis in a moving-towards-mixed-methods world needs to be put on analysis of qualitative inputs (and incorporation of those inputs into the larger whole). my hunch is that part of why people think planning for analysis of qualitative work may difficult is because, often, people don’t plan to ‘analyze’ qualitative data. instead, perhaps, the extent of the plan is to collect data. and then they plan to find a good quote or story (“anecdata”) — which may raise some questions about whether social science research is being done. not planning for analysis can limit one’s ability to set out a plan for analysis. this is different than saying that planning is not possible — there’s plenty of books on qualitative data analysis (e.g. here, here, here, and many others). here are some things that can be planned in advance:

  1. how will interviews and field notes be transcribed? verbatim?
  2. how will notes or transcripts be stored and analyzed? handwritten? in excel (my tendency)? using software such as NVivo?
  3. are you planning to be guided by theory or be guided by the data in your analysis? for example, maybe the domains you initially planned to study were themselves from some theory or framework about how the world works. this suggests that your analysis may be theory-driven and you may at least start with (promise to) close-code your data, looking for instances and examples of particular theoretical constructs in the transcripts and field notes and “code” (label) them as such.

or, maybe the work is far more exploratory and you set out to learn what people think and do, in general, about a particular topic. it’s more likely that you’ll be open-coding your data — looking for patterns that emerge (ideas that are repeatedly raised). and it’s likely you’ll have some idea in advance that that is what you intend to do. even if you start out closed-coding, it may turn out that a whole lot of your data end up falling outside the initially planned theoretical framework. fine. that doesn’t mean that you can’t report on what did fit in the framework (=plan) and then analyze all that interesting stuff that happened outside it as well. which, i think, is why are talking about pre-analysis plans rather than pre-analysis straitjackets. to close, in discussing whether pre-analysis plans for qualitative research — in the sense of talking to and watching living people, perhaps as part of a mixed-methods research agenda — are feasible, I hope the conversation is guided by whether planning is indeed possible in the social sciences as opposed to whether such planning currently takes place.

Doing research? Spend more time with program staff…

Originally posted on Suvojit Chattopadhyay:

From Duncan’s blog covering the paper ‘Using Participatory Process Evaluation to Understand the Dynamics of Change in a Nutrition Education Programme’, by Andrea Cornwall

While it is always interesting to hear about interesting approaches to data collection and analysis – and a lot of this feeds into thoughts/reactions I had while attending (by complete happenstance) the first day of the Doing Development Differently workshop at the Kennedy school, in this blog, I wanted to quickly highlight something that caught my eye in Duncan’s blog

Its this bit in particular:

“What we did next was not to go to the program sites, but spend some more time in the headquarters. We were interested in the perspectives of a variety of people involved with implementation, from the managers to those involved in everyday activities on the ground. And what we sought to understand was not only what the project had achieved…

View original 197 more words

small thoughts on transparency in research (descriptions of methods, analysis)

there is currently a good deal of attention on transparency of social science research – as there should be. much of this is focused on keeping the analysis honest, including pre-analysis plans (e.g.) and opening up data for re-analysis (internal replication, e.g. here and here). some of this will hopefully receive good discussion at an upcoming conference on research transparency, among other fora.

but, it seems at least two points are missing from this discussion, both focused on the generation of the analyzed data itself.

 

intervention description and external replication

first: academic papers in “development” rarely provide a clear description of the contents of an intervention / experiment, such that it could be, plausibly, reproduced. growing up with a neuroscientist / physiological psychologist (that’s my pop), i had the idea that bench scientists had this part down. everyone (simultaneously researchers and implementers) has lab notebooks and they take copious notes. i know because I was particularly bad at that part when interning at the lab.*

then, the researchers report on those notes: for example, on the precise dimensions of a water maze they built (to study rodent behavior in stressful situations) and gave you a nice diagram so that you could, with a bit of skill, build your own version of the maze and follow their directions to replicate the experiment.

pop tells me i am overly optimistic on the bench guys getting this totally right. he agrees that methods sections are meant to be exact prescriptions for someone else to reproduce your study and its results. for example, they are very detailed on exactly how you ran the experiment, description of the apparatus used , where reagents (drugs) were purchased from, etc. he also notes that one thing that makes this easier in bench science is that “most experimental equipment is purchased from a manufacturer which means others can buy exactly the same equipment. gone are the dark days when we each made our own mazes and such. reagents are from specific suppliers who keep detailed records on the quality of each batch…”

then he notes: “even with all this, we have found reproducibility to be sketchy, often because the investigators are running a test for the first time. a reader has to accept that whatever methodological details were missed (your grad student only came in between 1 and 3AM when the air-conditioning was off) were not critical to the results.” or maybe this shouldn’t go unreported and accepted.

the basic idea holds in and out of the lab: process reporting on the intervention/treatment needs to get more detailed and more honest. without it, the reader doesn’t really understand what the ‘beta’ in any regression analysis means – and with any ‘real world’ intervention, there’s a chance that beta contains a good deal of messiness, mistakes, and iterative learning resulting in tweaks over time.

as pop says: “an investigator cannot expect others to accept their results until they are reproduced by other researchers.” and the idea that one can reproduce the intervention in a new setting (externally replicate) is a joke unless detailed notes are kept about what happens on a daily or weekly basis with implementation and, moreover, these notes are made available. if ‘beta’ contained some things at one time in a study and a slightly different mix at a different time, shouldn’t this be reported? if research assistants don’t / can’t mention to their PIs when things get a bit messy in ‘the field’, and PIs in turn don’t report glitches and changes to their readers or other audiences, then there’s a problem.

 

coding and internal replication

as was raised not-so-long-ago by the nice folks over at political violence at a glance, the cleaning and coding of data for analysis is critical to interpretation – and therefore critical to transparency. there is not enough conversation happening about this – with “this,” in large part, being about construct validity. there are procedures for coding, usually involving independent coders working with the same codebook and then doing a check for inter-rater reliability. and reporting the resultant kappa or other relevant statistic. the reader really shouldn’t be expected to believe the data otherwise, on the whole “shit in, shit out” principle.

in general, checks on data that i have seen relate to double-entry of data. this is important but hardly sufficient to assure the reader that the findings reported are reasonable reflections of the data collected and the process that generated them. the interpretation of the data prior to the analysis – that is, coding and cleaning — is critical, as pointed out by political violence at a glance, for both quantitative and qualitative research. and, if we are going to talk about open data for reanalysis, it should be the raw data, so that it can be re-coded as well as re-analyzed.

 

in short, there’s more to transparency in research than allowing for internal replication of a clean dataset. i hope the conversation moves in that direction — the academic, published conversation as well as the over-beers conversation.

 

*i credit my background in anthropology, rather than neuroscience, with getting better with note-taking. sorry, pop.

To Get More Out of Science, Show the Rejected Research

Originally posted on Berkeley Initiative for Transparency in the Social Sciences:

CARL WIENS

CARL WIENS

In a recent opinion piece on the New York Times news portal the Upshot, Brendan Nyhan, an assistant professor of government at Dartmouth College, comments on a host of transparency related issues.

Closely echoing the mission of BITSS, Nyhan identifies the potential of research transparency to improve the rigor and ultimately the benefits of federally funded scientific research writing:

The problem is that the research conducted using federal funds is driven — and distorted — by the academic publishing model. The intense competition for space in top journals creates strong pressures for novel, statistically significant effects. As a result, studies that do not turn out as planned or find no evidence of effects claimed in previous research often go unpublished, even though their findings can be important and informative.

View original 335 more words

sherlock holmes and the princess bride

when i need to decompress, zone out, and fall asleep, i have a tendency to watch familiar, i-can-recite-by-heart movies or TV shows to help. just enough distraction to close down my other thoughts, not gripping enough to keep me awake.

the ‘sherlock‘ TV series has joined these ranks. the more i watch ‘a study in pink‘ (if i stay awake to the near-end)  the more i have to ask whether i, during the scene of selecting the good or bad bottle, could have resisted making the cab driver a sicilian (or making another princess bride reference).

props to the screenwriters on avoiding the temptation.