Thinking About Stakeholder Risk and Accountability in Pilot Experiments

This post is also cross-posted here in slightly modified form.

Since I keep circling around issues related to my dissertation in this blog, I decided it was time to start writing about some of that work. As anyone who has stood or sat near to me for more than 5 minutes over the past 4.25 years will know, in my thesis I examine the political-economy of adopting and implementing a large global health program (the affordable medicines facility – malaria or “AMFm”). This program was designed at the global level (meaning largely in D.C. and Geneva with tweaking workshops in assorted African capitals). Global actors invited select Sub-Saharan African countries to apply to pilot the AMFm for two years before any decision was made to continue, modify, scale-up, or terminate. It should also be noted from the outset that it was not fully clear what role the evidence would play in the board’s decision and how the evidence would be interpreted. As I highlight below, this lack of clarity helped to foster feelings of risk as well as a resistance among some of the national-level stakeholders about participating in the pilot. . . as  . .

To push the semantics a bit, several critics have (e.g.) noted that scale and scope and requisite new systems and relationships involved in the AMFm disqualify it from being considered a ‘pilot,’ though i use that term for continuity with most other AMFm-related writing. . .

In my research, my focus is on the national and sub-national processes of deciding to participate in the initial pilot (‘phase I’) stage, focusing specifically on Ghana. Besides the project scale and resources mobilized, one thing that stood out about this project is that there was a reasonable amount of resistance to piloting this program among stakeholders in several of the invited countries. I have been very fortunate that my wonderful committee and outside supporters like Owen Barder have continued to push me over the years (and years) to try to explain this resistance to an ostensibly ‘good’ program. Moreover, I have been lucky and grateful that a set of key informants in Ghana that have been willing to converse openly with me over several years as I have tried to untangle the reasons behind the support and resistance and to try to get the story ‘right’. . .

The set-up of the global health pilot experiment, from the global perspective, the set-up was a paragon of planning for evidence-informed decision-making: pilot first, develop benchmarks for success and commission an independent evaluation (a well-monitored before and after comparison) — and make decisions later. . .

In my work, through a grounded qualitative analysis, I distil the variety of reasons for supporting and resisting Ghana’s participation in the AMFm pilot to three main types: those related to direct policy goals (in this case, increasing access to malaria medication and lowering malaria mortality), indirect policy goals (indirect insofar as they are not the explicit goals of the policy in question, such as employment and economic growth), and finally those related to risk and reputation (individual, organizational, and national). I take the latter as my main focus for the rest of this post. . . . .

A key question on which I have been pushed is the extent to which resistance to participation (which meant resisting an unprecedented volume of highly subsidized, high-quality anti-malarial treatments entering both the public and the private sector) emerges from the idea of the AMFm versus the idea of piloting the AMFm with uncertain follow-up plans. . ..

Some issues, such as threats to both direct and indirect policy goals often related to the AMFm mechanism itself, including the focus on malaria prevention rather than treatment as well as broader goals related to national pride and the support of local businesses. The idea of the AMFm itself, as well as it a harbinger of approaches (such as market-based approaches) to global health, provoked both support and resistance. . . .

But some sources of resistance stemmed more directly from the piloting process itself. By evidence-informed design, the global fund gave “no assurance to continue [AMFm] in the long-term,” so that the evaluation of the pilot would shape their decision. This presented limited risks to them. At the national level, this uncertainty proved troubling, as many local stakeholders felt it posed national, organizational, and personal risks for policy goals and reputations. Words like ‘vilification‘ and ‘chastisement‘ and ‘bitter‘ came up during key informant interviews. in a point of opposing objectives (if not a full catch-22, a phrase stricken from my thesis), some stakeholders may have supported the pilot if they knew the program would not be terminated (even if modified), whereas global actors wanted the pilot to see if the evidence suggested the program should (not) be terminated. Pilot-specific concerns related to uncertainties around the sunk investments of time in setting up the needed systems and relationships, which have an uncertain life expectancy. also, for a stakeholder trying to decide whether to support or resist a pilot, it doesn’t help when the reputation and other pay-offs from supporting are uncertain and may only materialize should the pilot prove successful and be carried to the next stage. . . .

A final but absolutely key set of concerns for anyone considering working with policy champions is what, precisely, the decision to continue would hinge upon. Would failure to meet benchmarks be taken as a failure of the mechanism and concept? A failure of national implementation capacity and managerial efforts in Ghana (in the face of a key donor)? A failure of individual efforts and initiatives in Ghana? .

Without clarity on these questions about how accountability and blame would be distributed, national stakeholders were understandably nervous and sometimes resistant (passively of actively) to Ghana’s applying to be a phase I pilot country. To paraphrase one key informant’s articulation of a common view, phase I of the AMFm should have been an experiment on how to continue, not whether to continue, the initiative. . . .

How does this fit in with our ideas of ideal evidence-informed decision-making about programs and policies? The experience recorded here raises some important questions when we talk about wanting policy champions and wanting to generate rigorous evidence about those policies. Assuming that the policies and programs under study adhere to one of the definitions of equipoise, the results from a rigorous evaluation could go either way.

What risks does the local champion(s) of a policy face in visibly supporting a policy?

Is clear accountability established for evaluation outcomes?

Are there built-in buffers for the personal and political reputation of champions and supporters in the evaluation design?

The more we talk about early stakeholder buy-in to evaluation and the desire for research uptake on the basis of evaluation results, the more we need to think about the political economy of pilots and those those stepping up to support policies and the (impact) evaluation of them. Do they exist in a learning environment where glitches and null results are considered part of the process? Can evaluations help to elucidate design and implementation failures in a way that has clear lines of accountability among the ‘ideas’ people, the champions, the managers, and the implementer’s? These questions need to be taken seriously if we expect government officials to engage in pilot research to help decide the best way to move a program or policy forward (including not moving it forward at all).

learning by asking: a modest proposal to engage those who did the doing

this post was originally published at innovations for poverty action. it represents an effort to follow-through on some themes i have raised before and hope to continue to raise.

most development interventions are carried out and delivered by local research staff and residents. such implementation is rarely a straightforward ‘technical’ operation but, rather, there is social and political nuance in translating an idea into practice. on-the-ground partners therefore often have important insights about research engagement and implementation processes. but researchers often do not solicit such feedback as a requisite part of their work. moreover, even when such questions are asked, the results are often not synthesized into the lessons learnt from the project, either in terms of setting the future research agenda or in terms of tweaking on-going programs and interventions.

the Preserving ACTs (PACT) impact evaluation used text messages (via this code, built by @eokyere and @slawarokicki) to encourage the completion of the most effective anti-malarial treatment (artemisinin-based combination therapies, or ACTs (e.g.)) in and around Tamale, Ghana. as the study drew to a close, we wanted to thank the malaria-drug vendors for implementing the text-message intervention under investigation and for hosting the study’s surveyors. in addition, we wanted to solicit their feedback about the project and research process. our goals were to better understand their experience working with the research team and the study process, of the intervention process, and whether, how, and with what changes they thought the intervention should be scaled following the study.

photo credit: alidu osman  tuunteya

photo credit: alidu osman tuunteya

in mid- to late- 2011, one of our field managers went back to all 73 vendors – hospitals, clinics, pharmacists, and licensed chemical shops, both public and private – that had hosted our study, to thank them and to solicit their feedback on the study process and PACT intervention. 65 vendors were available and willing to be interviewed. these follow-ups took place in the third and fourth week of October 2011 and provide a unique set of data for considering the implementation of this and future studies. they also shed some light on the program that provided the context and basis for this operational research study – the Affordable Medicines Facility – malaria (AMFm) — discussed more at the end of this post.

as a token of our appreciation for their hosting PACT and our surveyors (and the only incentive provided to vendors for hosting the study and intervention), we provided a thank you note / certificate of participation. a few of the vendors noted that our follow-up was appreciated, such as “we thought you left without a goodbye” and  “I really never thought you would be back again after that long time.  But it is great you still remembered us and even come to officially say goodbye to us.”

as described in the paper, drug vendors played an important role in the study, which spanned much of the rainy season in northern Ghana as well as the Muslim fasting period of Ramadan, as mentioned in some of the follow-up interviews.

first, vendors were in charge of distributing the flyers to patients acquiring a malaria treatment. from this flyer, the patient would know to call or ‘flash’ (ring once for no charge) into our ‘mobile health information system,’ from which they were randomized to receive reminder text messages.

second, vendors provided an IPA surveyor a place to sit and talk to patients about their anti-malarial treatment and, if eligible, to recruit them into the study. Sometimes this meant giving them a chair in a hospital waiting room; other times, it meant the IPA team providing a chair for a surveyor to sit in the limited space offered by a one-man or one-woman drug shop.

given the study design, the distribution of flyers and the text messages were kept conceptually and, as feasible, spatially, separate from the IPA surveyor and his or her questions about the patient’s malaria medication. we included a question about this in the follow-up: the full range of vendor responses can be viewed here [link to the spreadsheet], as well as an overview of how responses were coded.* in short, the codes used to categorize responses emerged from the responses themselves in light of themes or ideas repeated across several answers.

given the sample-size needs of the study, surveyors were only placed at vendors that had ACTs in stock and were selling at a reasonable volume. this often meant longer stays at some of the hospitals, clinics, and popular pharmacies and LCSs but shorter stays with other vendors. some of the vendors expressed disappointment about this, such as, “I felt the surveyors did not spend much time in my shop.  You should have stationed a surveyor at my shop for at least three weeks or 4 weeks” and “I did not experience much due to a lack of patients during the study period at my shop.”

experiences with PACT.

though we did not explicitly ask vendors if PACT addressed a problem they thought was important, three-quarters volunteered that non-completion of ACTs was a problem they faced and many welcomed the support of a program like PACT to try to encourage people to finish their medication and to strengthen whatever advice they already gave to their patients – for example, saying that the text messages “made patients to take my advice more seriously” and that they “increase our credibility and patients to stick to these advice we give as a health service provider”.

others felt that the text messages led to patients feeling they were receiving addition concern for their wellbeing. one noted that PACT “gave patients the courage to come back to us and ask questions of the drugs given to them, and then some do come back later to report other issues” and another that “patients that did initially not want to participate have come later to commend the text reminder.”

prior to agreeing to host PACT, some vendors were concerned that the flyer and survey process would interrupt the patient flow, affect their business, or lead to an over-focus on malaria patients at the expense of other clients. after the study, about 40% felt their sales stayed normal during the study while about 20% felt their sales had increased, which they attributed to PACT. Nobody reported a disruption in their sales. about 40% did note that the questionnaires administered by surveyors were long and were uncomfortable for patients who were sick and tired – hence, being at a drug vendor in the first place.

about 20% also felt their status or credibility increased when they were offering the program, such as “business was normal but on social grounds, I earned respect from individuals that were sent SMS…  It was a remarkable experience of the concern they achieved from your end. So, I gained more credibility to my advantage.” some of these benefits may, of course, reflect a novelty effect and would not offer any one vendor an advantage if the program scaled-up to all vendors.

there were also a few unexpected benefits of the program likely not to be replicated in a scaled-up version of the program — for example, one vendor reported that “your surveyor also helped me track a thief that came to steal from me”!

several vendors highlighted that it’s not always easy for them when a study like this – which facilitates their providing an addition service to patients and clients — ends, noting, for example, that “some patients still ask me if they will get the SMS” and that “now I have to explain to my clients that I cannot send SMS to remind them now, which is sort of sad”.

continuing and scaling-up program.

most vendors (97%) explicitly said that it would be good the PACT program – or something like it – could be extended beyond Tamale and the time period of the study but also to other medications. for example, one noted “it should be scaled up so that at least I know my family that does not reside here in Tamale would also benefit from this great experience and education” while another said “the program should be nationwide because I know this attitude” – “the bad attitude of not completing the dosage” – “is widespread” and another similarly stated “the issue of people not completing their dosage is nationwide, so the program must be nationwide”.

about half saw no predictable barriers in being able to continue a program like PACT on their own; nearly a quarter thought that during busy times, it might be hard to make time to explain the program to patients and enroll them. 42% said they would need more training. Half noted that materials (such as flyers) and money for credit would need to be provided to them and several noted that extra components, such as t-shirts, would help motivate them and advertise the program better. for example, “I would also love to be given [something extra], like t-shirts, that could help motivate the vendors to do the work diligently.”

vendors also had a variety of suggestions on how enrollment into the text messaging system could be encouraged. One set of suggestions including removing any costs to patients, such as making the enrollment line toll-free as a way of encouraging more patients to enroll. to clarify what happened in practice, if patients texted into the program, their phone balance was deducted the cost of a text message. for people to ‘flash’ (dial and allow one ring) into the system, it did not cost them any money but phones (rather than our messaging system) required a positive balance to make this call.

more generally, 37 vendors (60%) suggested ways of increasing enrollment into the text messaging program, including the vendor or PACT team doing the actual enrollment as well as providing more information to patients about the program and why it was important. for example, one noted “if people are educated well, they will charge their phones and enroll later.” given the goal of the study — to see whether and how many people would enroll with a limited amount of encouragement and input — vendors provided limited (and surveyors provided no) information to the patient along with the flyer. in retrospect, however, some vendors saw this as a missed opportunity to educate patients about an important problem.

fourteen of the vendors raised concern that the text messages excluded illiterate patients – some suggested using voice calls as well as TV and radio to better educate about the importance of completing an ACT once it is started.

distinctions with AMFm (and implied suggestions for future AMFm-like programs).

the Affordable Medicines Facility – malaria (AMFm) provided the context for the PACT study, although the text messaging program was meant to be seen as independent of the ‘green leaf’ efforts explicit to AMFm marketing. as such, we asked questions of vendors to assess whether they saw the programs as distinct. In so doing, they ended up commenting on differences in the program in a way that may shed light on future AMFm-like efforts. for example, “programs for the ‘green leaf’ have more TV and radio coverage than your PACT program, but then you made more impact of encouraging people to take their drugs.  The other programs only encourage buying and awareness” or that with AMFm, “people do not get education on usage or encouragement to complete dosage”.

to clarify, the AMFm-linked advertisements focused on telling people to act fast when they suspected they had malaria, to look for the “green-leaf” quality logo, and what price these logoed ACTs should cost. as such, one vendor noted “the ‘green leaf’ programs don’t have any sort of attachment to the patient but your intervention seemed to involve and appreciate the patients more, so they felt respected and began to place some emphasis on their health.”

also, it is worth noting that elements of PACT seemed to alter the way patients viewed ACTs as well as the perceptions and stocking decisions of the vendors themselves. overall, 65% of vendors reported such changes. as part of AMFm, vendors should have received trainings provided by various government and professional bodies, as well as the mass media “green leaf” advertisements — but this seems to have not always been the case, at least by the time the PACT study was carried out.

for example, one vendor noted that “people are now more confident in the ACT drugs” and that “I now keep more stock of ACTs and on days that I do not have ACTs, the clients disturb me so much.  So for now, I make sure to keep more stock of ACTs.” Another said, “to be honest, it was the start of your study and the subsequent positioning of a surveyor at your shop that made me to purchase ACTs, and now I have many varieties of ‘green leaf’ drugs” while another reported “we were a bit skeptical about the new ACTs, so we had few in stock, but your presence also made us want to be more serious and also keep different types of ACTs.”

overall, the data presented above and in the spreadsheet (available below) suggest there is much that can be learned from soliciting feedback from program implementers, which can be taken into account in designing future studies and interventions. this type of engagement can ensure that humanity and humility remain part of the implementation and learning process.

you can download the spreadsheet with more information and responses here (excel format).

*where a vendor directly identified themselves or their workplace in their response, we have slightly altered the response, using italics in place of the missing word or phrase.

**important thanks go to Jeff Mosenkis, Corrina Moucheraud, Elianne Oei, John Quattrochi, and the late Elif Yavuz for encouragement in writing-up and thinking-through these results. the data and photographs above were collected by Usama Salifu and Alidu Osman Tuunteya. the biggest thanks go to the vendors for taking the time and energy to provide additional insight.

here’s an idea — if they are trying to tell you something, make it easy for them to do so.

there’s been a good deal of press around the unfortunately insignificant results of a major HIV prevention trial with products for women in south africa, uganda and zimbabwe. the results had little to do with efficacy of the products (a pill and a gel) but rather with the fact that most of the participating women did not use the treatments as recommended – or at all.

one potential response is to improve our behavioral interventions to support adherence to treatment regimens (and prevention regimens) and integrate these methods more directly into medication trials. adherence and persistence with medication are global problems and we are just beginning to learn – with the help of health psychology and behavioral economics – how to tackle the challenge. efforts so far include high- and low-tech solutions, though not all the promises of the former, in terms of mhealth to facilitate behavior change and adherence, have yet been borne out.

another, not mutually exclusive, response would be to actually ask the women what they would like to see and use in the way of HIV prevention – a tool which should be empowering for them. the press seems full of comments like “the women are trying to tell us something!” why does it seem that, then, for a product made for them, they have to work so hard to tell us those things? why are we not hearing more sentences that start “the women told us…” that is, why, after such a big trial, am i not hearing anything about on-going qualitative and observational follow-up efforts to learn more about what exactly didn’t work about the methods offered to women?

there’s often a lot to learn from null results (that’s science, right?) but it doesn’t just come from brainstorming what went wrong. asking helps.

i don’t suggest that people are perfectly prescient about what they need or want. often, the innovations that we can’t live without now – smartphones, for example – weren’t a need or even desire that most people could have articulated 20 years ago. as such, directly asking people what it would take to get them to engage in X desirable behavior can’t determine the research agenda. but it should certainly be part of figuring it out.