While i have given some account of my symptoms in earlier posts, it seems useful to give a quick recap to contextualize why i found the hospital space so difficult to navigate. And let’s be clear from the outset: i am an extremely privileged patient in Delhi. i am physically inclined to get good service in hospitals; i can read and, moreover, can read English, the language of most hospital signs (at the hospital chain i attend, most signs seem to be in English only except for the one about not doing pre-natal sex determination); and i have enough medical knowledge and confidence to ask my doctors questions. My navigation issues were not about lack of comprehension but rather poor design and poor service.

.

Though i had been feeling grungy for a few days, the distinct symptoms set in on a Monday morning. My alarm went off and i woke up, ready to push myself into a sitting position in bed, only to discover that my wrists and elbows were not interested in supporting my weight. On later mornings, they didn’t want to bend from being locked down by my sides and i would have to spend many minutes consciously willing my shoulders and elbows to cooperate so that i could get my hands to my face to wipe the sleep out of my eyes with my thumbs, since the rest of my fingers were stuff and against the idea of balling into fists.

.

But that first morning, having had my arms joints turn to jelly, i tried to push the sheets off of me, only to discover that my fingers were not interested in pinching the sheet to move it. No doubt this all looked a bit funny, a bit like the Wolf of Wall Street quaaludes scene, but it is worth noting that it was actually terrifying. it is worth noting this because it is something that not a single doctor acknowledged over the course of this illness. Doctors: i don’t care how many patients you have coming in each day complaining about the same symptom set — overnight loss of functionality is scary to each of those patients and deserves to be acknowledged as such.

.

In any case, on the first morning, i finally did some worm maneuvers to at least get myself sitting up and ready to try standing. This is when i learned that my legs and feet weren’t faring much better. The arches of my feet felt like they couldn’t support my weight and the rest of my toe joints didn’t want to bend to facilitate walking. My knees had gone to rubber while my hips were like the Tin Man in Oz. It was a long, slow, fairly ignoble shuffle to the bathroom. Once reaching, i learned that my fingers weren’t going to support gripping a toothbrush, a water bottle, a hair brush — really anything at all. My thumbs would work for texting but not much else.

.

Just like that, i became a 90-year-old woman (crankiness included).

.

When i finally got back to bed, my only interest was trying to get joints to pop, which seemed to hold the elusive promise of some pain relief. Besides napping, i am not sure i did anything the first day besides bending and extending my right leg in hopes of a knee pop and doing the same with my left elbow, which stubbornly refused to cooperate.

.

All of this is to say that i wasn’t in much of a shape to walk by the time i decided to get to the doctor the next day — forcing me to recognize explicitly just how much mobility is expected of patients in a hospital. To leave my flat, i navigated the marble stairs down, relying mostly on weight put on my forearms on the railings, since i had as little faith in my wrists as i had in my ankles to support me. My mostly faithful autowalla Pankaj drove mercifully slow and cursed out another driver that tried to rush him.

.

At the hospital branch (of a larger corporate chain) nearest my house, there isn’t a real drop-off and pick-up space, since the hospital is sort of nestled on a residential ‘slip road’ that runs parallel to the main road. This creates traffic jams and issues. Nevertheless, Mr. Traffic Mover, don’t yell at sick people to get out of the car faster or to walk faster. Of course, not all people who go to a hospital/clinic are sick and not all of those who are sick have mobility issues — but let’s make some basic assumptions about the types of people who are arriving to a hospital and try to accommodate.

.

Walking to get into the clinic is hardly the only walking that you will need to do — indeed, i wondered multiple times if the the architects of clinic and patient-flow systems had ever considered the fact that patients might be sick in their design.

.

First you go to a counter to get a token in order to go somewhere else to sit and wait to pay for your consultation. Then you’ll pay for your consult (also, attendants/receptionists, if someone asks for help getting their credit card or cash out of their wallet, bloody help them. Again, you work in a hospital, where people need help).

.

Then you’ll go off to find, first, the queue at the ‘sisters’‘ station (though it is a secular hospital, the nursing aides are referred to as sisters) to get your vitals taken (where you may have to wait), then probably walk somewhere else to wait for your consult and then walk to the consult room itself. None of these are short distances around a central waiting room; some of the trips involve changing levels.

.

You may also get lost along the way, since there are at least two ‘sisters’‘ stations that look identical but are on opposite ends of a large open space. Since the receptionist could only muster enough energy to vaguely wave that you need to go somewhere behind her, you don’t know whether to turn right or left. The signs indicate that both directions are legitimate options. Sick people, in general, are trying to conserve steps and don’t particularly appreciate getting themselves to one station only to be told that they should have gone to the identical looking one that offers the identical reading of blood pressure and temperature — but is located on the opposite side. Give reasonable instructions, as though you are trying to minimize the effort the patient needs to make.

.

If you are prescribed a blood or other test, you make your way back to (wait in line for) the front reception to pay in advance for the test, then go off to find the samples collection station. And if, say, you were only charged for some but not all of the tests prescribed to you, you will have to walk back to reception (where you will not receive an apology) to pay and then back to the sample collection station.

.

Then you can make your way to the pharmacy and finally, attempt to leave, after spending several minutes cajoling the fellows standing outside at a podium to help you flag down an auto.

.

Surely there is way to streamline some of this back and forth or at least make it incumbent on hospital staff, not the patient, to do the some of the running around. There are, so far as i could tell, plenty of staff idling about.

.

At one particularly low moment, i found myself in the west block (separate building) trying to pick up two sets of lab results, one taken in the west block and one taken in the east on two prior trips to the doctor. At first the receptionist explained that by entering my mobile number and patient ID (which apply throughout the whole hospital system, at least in order to make an appointment online), she would only be able to print the results from the west block sample and that i could walk over to east block to have someone put the exact same information in a computer over there and print out the other set of results. This is the moment when i lost my cool. As it turned out, yes, one of the idle staff could go over on my behalf and claim the other set of results.

.

The question is: why, as a pretty clearly unfunctional person, i had to ask for basic assistance. Seriously, it’s a hospital, people are sick, they need help, and not every patient has 10 family members with them. Design your floor plan and your services to accommodate that.

.

For better or for worse, focus group discussions are a common way of gathering qualitative information in the social sciences, as a standalone activity or used in parallel or sequentially with other types of research activities. My general sense is that focus groups are used far too often when one-on-one interviews [do not use FGDs to attempt to try to ‘efficiently’ do many individual interviews at once] would have been more appropriate to the information sought (and the analysis done, if indeed it was done at all).

.

This last point is important: certain ways of gathering qualitative data are more appropriate to gaining certain types of information (for example, are you collecting sensitive information? Are you actually relying on the power of group discussion and reflection to generate useful insights or are you trying to conduct several (slapdash) individual interviews at once?).

.

For interpretation, it is quite helpful if researchers explained and justified their choice of qualitative data collection strategy given their research aims. (Surely appropriate justification and description of methods is part of the research transparency agenda, regardless of whether we are discussing quantitative or qualitative data?)

.

More generally, as part of my work with IDinsight, i have been putting together a guidance document on making apt use of qualitative data and qualitative approaches to research for the aims of evaluation and learning. It is still very much a work-in-progress but a small sub-section is “stuff you should about data collection include in your write-up if you opt to use focus groups.”

.

Here’s my working list of what i would want to know about your data collection activities in order to render your presented results from group discussions far more interpret-able, believable, and useful in informing future work [note that much of this can be written up before data collection begins and could even be part of a commitment to (pre-) analysis & reporting plan! Deviations from the plan could also be reported after data collection is complete]:

.

Objectives, justification, sequencing

• How do the focus group discussions fit in the overall research strategy? What informed the questions asked (theory, a prior research stage, etc)?
  • For example, is the work being done to inform the intervention design or is it being used to understand a previous result?
• What were the main questions/topics/themes intended to be covered in the groups and were these meaningful in the given overall research aims? (Please say something more useful than ‘to add context or color.’)
  • Was the intention to be hypothesis generating, hypothesis testing, or both? Clarify if different topics/themes were pursued for different purposes.
  • Was the intention to be inductive, deductive, or both?
• Why were focus group discussions selected as the data collection strategy given the research objectives (and practical constraints)?

Positionality and placement

• Who (in the sense of positionality) was part of the moderating team (such as a facilitator, a note-taker, and an interpreter if relevant)? Why did this make sense given the research setting and questions? Was the team the same for all groups conducted? If not, what dictated the differences?
• Where were focus group discussions held? (this can be general, such as ‘a community space was identified in each village with the help of a local elected official’)? Was the space sufficiently private and free of distraction in most or all locations?

Composition and other group details

• Who comprised the group (in a general sense, such as ‘women with children’)? How were these people selected? Did people in the groups generally know each other or not? Why was this composition strategy seen as the appropriate composition given the research objectives?
• If different compositions of groups were sought (such as interviewing men and women separately or program participants and non-participants separately), why did this make sense in this context for the given research objectives (and how did it connect to a broader theory of change and/or expectations about heterogeneous experiences)?
• How many groups were conducted all together? Why was this seen as a sufficient number? (Ideally, the answer will include some reference to saturation on key themes.)
• What as the minimum, mean, and minimum size of all the groups conducted?
• What was the minimum, mean, and maximum duration of all of the groups conducted?

Iteration throughout data collection

• Did the goal of the focus group discussions change over the course of the work? If so, how and why?
• What main changes were made to the interview guide / key topics explored over the course of the data collection? (And, if not obvious, why were these changes made, such as being too contentious or an unconsidered angle emerged?) Any key lessons about questions that ‘worked’ or did not?

.

More to follow on details to report for data organization and analysis, including elements of capturing power dynamics during groups… In the meantime, thoughts welcome on this list. i’m updating the list as ideas come — so far, many thanks to @urmy_shukla and @_alice_evans for some additional ideas (added 19 Sept 2016).

 One of the most interesting things i have noticed over this illness episode, though it’s been present throughout my three years in Delhi, is how accessible doctors are expected to be here – and how easy it is to fall into a habit of WhatsApp-ing (now officially a verb?) with your doctor (and sometimes getting a ‘thumbs up’ emoji in reply, which is oddly validating after you report that your fever has broken or some other milestone has been reached — even when you know your doc may not actually know with which patient s/he is communicating, something i learned later!).

.

To begin this potential relationship, you receive (at least one of) his/her mobile number on the paper you get with the notes on your consult as well as the prescription (one of the many pieces of paper one accumulates).

.

i am hardly alone in availing the doctor’s number, as evidenced in part by the number of calls or texts a doctor might take during your 10-minubte appointment slot: clarifying prescriptions, checking whether a clinic visit is necessary for certain symptoms — and so on. This disruption is compounded by nurses, patients and others coming in and out of the consult room. A closed door has very little meaning — or at least not the meaning i expect it to have. (The curtain around the examination table is, mercifully, more sacred.)

.

As indicated, i now WhatsApp fairly regularly with one of my doctors, which was quite comforting when, a day after one of my appointments, i woke up from a nap with a fever and a full-body rash. i was able to text her and received a response to send a picture of the rash. this raised some new and interesting questions:

• Is it easy to operate a camera/phone when your fingers are arthritic? (answer:no)
• What is the best lighting to capture a good picture of a rash? (answer: still unclear. all my pictures looked like everything was red, which was only partially true)
• Is there a good way to get a picture of a full-back rash without ending up with something risque to send to your doctor? (answer: i couldn’t do it successfully, so these pictures did not go)

.

These more informal chats also facilitate meeting outside the corporate private hospital. From studying health systems, i know that these sort of evening and weekend visits at small, private clinics are in many places a common feature of the public health system (government doctor by day, private practitioner by night). This was my first experience of big-private by day, small-private on the weekends. There were several pluses to seeing my doctor at her family’s imaging center rather than in the big hospital — lower consult fees, longer appointment time, more flexible appointment time (i.e. the morning after the rash broke out), less crowded. Mixed in with some guilt of being in a waiting room mostly of expectant mothers awaiting ultrasounds while i am blatantly feverish.

.

As a small vignette from this particular visit to the imaging center, at about the lowest point of the illness (limited mobility + fever) there was a gap between when the doctor finished her consult and when i could get a particular needed scan done. The doctor let me nap on the exam table (no curtain) while she preceded with another consult in the same room. No one else seemed fazed and i certainly didn’t care by that point.

.

Another manifestation of this tendency towards convenience and connection is the wide availability of diagnostic services that ‘come home’ (or ‘come to office’). In any case, diagnostic and imaging centers are quite often standalone entities, separate from the doctor’s practice (which also means you can walk in and ask for pretty much any test you want). Now of the big chains have set up mobile services. i have yet to avail this as someone pointed out that i may not want be a(n ill) woman home alone and invite someone in to my house to take my bodily fluids — but everyone who has tried this seems to find the service quite professional as well as convenient.

.

When at a pharmacy last week for my ever-changing cocktail of painkillers, i noticed a sign alerting me that they can come home to do full cancer screens for the whole family. Which, (a) some things still feel like they should happen in a clinical setting — but perhaps i am old-fashioned? and (b) what a dreadful family bonding activity.

.

In terms of communications with doctors and getting clinical services outside the clinic, i presume in Delhi, i am witnessing both the past and the future. There is still a tradition/expectation (among the middle- and upper-classes, in any case) of a family doctor who lives near by, who is also a family friend and might come over for chai and check-up and is available by phone when needed. So (working hypothesis) the past is setting present expectations and practices around access to some extent. But, if diagnostics can be untethered not just from the clinic but from brick-and-mortar in general (at least for collections, and then reports emailed to you), i suspect we might start to see this become a far more common practice globally.

i had the pleasure of giving two talks through 3ie seminar series over the past few weeks, in london (chaired by the wonderful daniel philips) and in delhi (chaired by the great stuti tripathi and ably discussed by the always-exciting colin bangay). i was also able to present at the london school of hygiene and tropical medicine — big thanks to catherine goodman and clare chandler for arranging and for great conversation. many thanks to everyone who participated (and everyone has listened about my thesis along the way). [slides for the interested.]

.

the presentations were on aspects of my thesis research, which centers on the experience of ‘doing’ the affordable medicines facility-malaria (amfm) in ghana. as often happens, the global narrative around the amfm and the decision to pilot-and-evaluate-and-decide was intriguing and became an important point of discussion. indeed, the pull of the story itself was a major drag on my getting on with aspects of my thesis (with apologies to my committee). what i present below is less about my actual work (relating more to national stakeholders opting in to the pilot and the implementation of it) and focuses on the global narrative, which is abbreviated and stylized in my telling.

.

i set aside the precise definition of equipoise that is most relevant in social science and development evaluations –- whether we should be talking about clinical equipoise (or efficacy equipoise?), some flavor of policy equipoise (or (relative) cost-efficiency uncertainty), or even operational equipoise (uncertainty around whether this thing can actually be implemented in this context, let alone produce the intended results).

.

rather, i begin with the assumption that meaningful uncertainty is a good starting point for commissioning an evaluation and also possibly a key part of the ethical justification for particular approaches to evaluation, such as random assignment. the former is of more interest to me here.

.

an important question – one that the amfm raises pointedly — following from this interest is what obligation, if any, follows from the establishment of equipoise (that a community of thinkers and/or implementers have meaningful uncertainty about a proposed program on a theoretical or practical level).

.

let’s look at the amfm. the goal was to make use of existing global and national public and private sector supply chains (from pharmaceutical manufacturers to small pharmaceutical sellers) to dramatically increase access to high-quality antimalarial treatment and, in turn, improve the (appropriate) use of such treatment and reduce the malaria burden.

.

this generated a situation of ‘active equipoise’ (read: sometimes heated controversy). some was more ideological: should the private sector be used to deliver public health, for example. i set this aside here. some were practical: if we use this specific mechanism to deliver a health commodity, will the subsidies (‘co-payments’) involved be captured along the supply chain or passed on to the end-user? will people not only obtain the high-quality, recommended anti-malarial treatments once they are made more accessible but use them (appropriately) and finally reduce the malaria burden?

.

given this degree of uncertainty about putting a theoretically ‘elegant’ (an oft-used epithet for the amfm) mechanism into practice, a decision was taken to pilot (at national-scale, in 7 countries, for 1.5 years, so the application of the term ‘pilot’ is debatable) and to commission an independent evaluation that would inform the decision to continue, modify, scale, or terminate the initiative. specifically, the global fund agreed to host the initiative for this pilot period and the evaluation was intended to inform the fate of the amfm, at least in the global fund’s portfolio. i am not going to wade into the confusion about the decision was ultimately made (also here) because i want to focus earlier in time than that, about the design of the evaluation itself given its intended decision-informing function.

.

note that there were three key points of (less-ideological) debate at the global level that prompted the pilot-evaluate-decide approach, that can plotted down a theory of change and also down a supply chain:

• implementation feasibility and the possibility of supply-chain capture (and drug adulteration)
• the translation of access into (appropriate) use
• the translation of use into reduced malaria burden

.

before going on, please note that i am not arguing that all evaluation or research needs to lead to a decision or even have this as a goal. rather, i am asking, once it is determined that we will commission an evaluation to inform our decision -– a pinnacle of evidence-informed decision-making — what are our (researchers, evaluators, funders, decision-makers) obligations (ethical or otherwise)?

.

for a variety of reasons, the global fund decided that they wanted the pilot to 1.5 years and, following from this decision, set four benchmarks deemed achievable (through modelling work) within that timeframe that would define success. these related to gains in availability, price, market share (gained against less effective but cheaper and more familiar anti-malarial treatments), and household use. even though the link between use and malaria burden was a key point of uncertainty, this was determined to be beyond the scope of the evaluation from the outset (which people might agree or disagree with). at some point in the process, household surveys were also dropped from the evaluation plan, also cutting off the potential to make rigorous (or, really, any) statements about whether access translated into use.

.

a result of this, it seems, is that many global stakeholders have been able to use the results of the independent evaluation (which suggest that at least in 5 of the 7 pilot countries, moderate to high success in access was achieved) to support whatever position they had initially. (the story at the national level seems a bit different: whether because of experiential learning or the evaluation results or path dependency or other factors, many national-level stakeholders seem to have wound up more supportive of the initiative than they were initially –- something which warrants further investigation.)

.

a key question is how we should feel about the issue of the evaluation – again, explicitly intended to inform a decision – not being set up to address the key points of controversy. disappointed? angry? ethically outraged (note, to the extent that money and not just principle matters, that this evaluation had a $10 million pricetag and that the overall piloting process rang in around $460 million)? this issue of appropriateness and outrage was a key point of discussion, particularly in the delhi seminar.

.

i certainly don’t have an answer but the question merits further debate. if an evaluation is commissioned to address specific points of controversy (uncertainty, equipoise) and explicitly to inform a decision, what are the obligations and responsibilities (whether practical or moral):

• of the evaluation design to address the controversy (in a way meaningful for those identified as key stakeholders or decision-makers)?
• to use the evidence generated to make the decision? (and to put in place processes to help make this so)

.

for those of us that push for evidence to play a role in decision-making, these seem important questions to debate. i hope we start to.

this is a joint post with urmy shukla.

.

in this blog, we advocate the importance of in-depth reporting on implementation processes, evaluation processes, and relevant contextual details of interventions and linked evaluations. this will facilitate research transparency, as well as assessments of both learning and the potential for generalizability beyond the original study setting (learning lessons from ‘there’ for ‘here,’ but not necessarily promoting the strict and exact duplication of a program from one setting to another, in line with an understanding of external validity that is appropriate for the social sciences in development).

.

we start with a hypothetical scenario of an intervention and associated evaluation, based on too-frequent experiences in the impact evaluation space. we hope that it doesn’t sound familiar to those of you who have been involved in evaluation or have tried to make sense of evaluation results — but suspect that it will.

.

a research team, connected to a larger research and evaluation organization, ran a study on an intervention. for reasons of statistical and political significance, they have deemed it sufficiently successful and worthy of scaling up, at least in a very specific new setting.

.

the intervention sought to overcome the following problem, for which there are supply-side and demand-side issues. people in malarious areas may procure a bednet (whether for free or for a positive price), but they do not always follow-through with maintenance (re-treatment or replacement).

.

for supply, the private sector only sporadically offers retreatment and replacement, and it is expensive, while the public sector does not always have supplies available. the intervention, therefore, concentrates provision of this service at a specific time and place through temporary service centers.

.

for demand, people with nets often don’t understand the need for retreatment and, even if they do, continuously put off doing so. the intervention, therefore, included a non-monetary incentive for which there is local demand (in this case, soap) to be picked up at the time of net retreatment.

.

this is, altogether, a relatively straightforward (or ‘technocratic’ in woolcock’s classification) but potentially powerful intervention that can improve the private and public good. as such, researchers in the research organization would like to try this intervention (with associated impact evaluation) in other locations, in which they suspect net retreatment and replacement faces a similar set of challenges.

.

however, when the research team in charge of the external replication looked back to the original reports from this experiment, they discovered relatively little information about how the intervention was designed and implemented. the publication is mum on the process itself and any lessons learned, including challenges faced and whether the researchers have done something different with the advantage of 20/20 hindsight. moreover, there aren’t many internal notes that lay out the operations of the intervention or the evaluation. what exists as tacit knowledge remains kept for elite seminar discussions or cathartic gossip over beers.

.

this hypothetical raises two key problems: (1) research transparency and (2) the potential for learning and assessing generalizability.

.

research transparency: from implementation to data collection

while the current focus of research transparency movements (across and within disciplines) is clear about making data and code available for internal/statistical replication, there is a critical piece missing about process. how was the evaluation run? how was the intervention run? what challenged and enabled the success of an intervention in a particular setting? from the hypothetical scenario, this includes questions such as:

• who was supposed to organize and run the service centers and who actually did so?
• did the evaluation put in place any kind of monitoring that would likely to be present if the implementers were acting alone? Should this be properly considered part of the intervention?
• how was the procurement of soap supply managed and were there any relevant challenges?
• how was soap determined to be a good incentive in this setting in the first place?

.

the research team should ideally be able to refer to a report or working paper, or at least set of internal notes to guide them. but a lack of documentation means that neither evidence users nor even those within the research organization know the answers to these questions. this isn’t just an issue of operations and redundant work, but one of research transparency and ethics: to understand what an intervention actually included, and what is required in order for it to be successful.

.

understanding the intricacies of both the implementation and study setting should include both systematic documentation of relevant factors (ideally informed by a theory of change), as well as ensuring that both quantitative and qualitative ‘process’ data are collected with the same rigor as ‘evaluation’ (baseline/endline) data. going beyond a bare-bones theory of change (and including theoretical mechanisms, implementation processes, and contextual interactions) requires extra work. this should, admittedly, fall with both researchers and donors/commissioners — to ensure that study teams have the necessary financial resources, time, and research capacity to effectively and systematically collect and process this information.

.

reporting for learning and generalizing

the ‘active ingredients’ of programs (if not whole programs) tested in one setting can also be tried in other settings (other times, geographies, scales, etc). indeed, some may say this is a key goal of policy-relevant evaluations. such trials may be done on a one-off basis or as part of a more systematic approach to external/field replications to learn whether some interventions are indeed effective in a variety of settings.

.

neither can be done well if details about how the active ingredients were implemented and measured are not reported. this, in turn, is quite difficult without tools to measure and document processes and decisions made along the way. but this needs to be sorted because reporting on implementation and evaluation experiences and challenges is central to a learning agenda in the social sciences and in programmatic and policy work.

.

this argument is not new. lincoln and guba call for “narrative developed about the [setting to allow] judgments about the degree of fit or similarity may be made by others who wish to apply all of part of the findings elsewhere.” it seems that similar concerns motivated woolcock to close his paper on external validity with a call for more case study work. thickness and richness of description, rather than thinness, helps users of evidence learn and make assessments and adjustments in light of their own setting. this description, guided by a good theory of change, can address directly some key challenges to external validity, such as site selection and partner selection biases.

.

in sum

failure to prioritize intervention details that track along a detailed theory of change could be potentially detrimental, with ill-advised implementation and/or a locally inappropriate intervention. in the case of the bed-net intervention, failure to report on challenges (e.g., need for extra community buy-in, developing efficient supply chains, clarity about what government workers could not handle alone, etc.), as well as enabling factors (e.g., community-level awareness of proper bed-net usage, the operational strength and local reputation of the implementing partner, etc.), could mean that the research team team conducts a study (and intervention) that, at the least, inefficiently uses both research and implementation resources, and at the most, has negative unintended consequences. ultimately, the conversations that combine research transparency and policy recommendations should prioritize high-quality, systematic, and readily available data from all parts of the impact evaluation cycle.

.

our current efforts to be ‘rigorous’ while conducting evaluations are insufficient. we must also be rigorous in our efforts to document evaluation and implement processes, to report on these, and to critically and openly reflect on how we might approach the same problem differently in the future. this level of transparency, though initially daunting, will only improve the potential for better-informed and better-implemented policies, aiding us in transferring lessons from ‘here’ to ‘there’.

when first introduced to the idea of a funnel of attrition (my early attempt at a slightly more nuanced and symetric — but still generic — version is here), i largely thought of it as a useful heuristic for thinking about sample size calculations, by being forced to think about issues of awareness and take-up as well as a few steps along a causal chain between initial participation or use and longer terms outcomes of interest.

.

more recently (including here), i  have tried to use it as a tool for thinking about articulating assumptions in a theory of change about where people might ‘fall out of’ (or never join) an intervention, thus leaving the funnel. more specifically, i tried (along with colleagues) using it as a goal for a conversation with implementing partners (that is, “let’s map out the funnel of attrition”), tackling the question from multiple perspectives. (various perspectives were brought in using personae, which i created beforehand relying partially on average results from the baseline as well as some stylizing to try to bring certain features into the conversation. at first i feared being overstylized but, in the end, i think i had too little detail. i reviewed my notes from the inmates are running the asylum and was reminded of the importance of specificity, even at the expense of accuracy.)

.

i liked this idea for guiding a conversation because the funnel of attrition is a little more straightforward than a full theory of change but, in constructing it, you still end up articulating some central assumptions, which can be added to thinking about change may/not happen. it seems like a handy building block in a well-considered theory of change.

.

the intervention in question is a multi-year engagement, with escalating levels of commitment required and difficulty of requirements. it initially struck me that given all the different program elements, it would be relatively easy to discuss about some of the different points at which it could become difficult to continue to stay engaged. this turned out not to be true. i played devil’s advocate far more than i had initially planned, effectively interviewing the personae with pretty specific questions that ended up reflecting my own thinking about the difficulties of sustained engagement more than revealing so many new ideas. this doesn’t mean the exercise was useless in getting others to think through these issues but i had certainly imagined it going a little differently.

.

my revised thinking is that it may help to have an early draft of funnel of attrition to guide the conversation, even if (or hoping that) this gets trashed and remade in the discussion process — it just helps to have something tangible. this draws on an interesting suggestion from steve montague of even building in a few ‘errors’ into an early draft understanding of a theory of change to present to implementers to help provoke discussion. my working plan for the future engagements i’ll do with the same implementing team is to walk through this draft funnel of attrition from multiple, quotidian perspectives, with specific thought about factors that might facilitate, reinforce, or hinder progression down the funnel (meaning, staying with the program and realizing the intended changes). 

.

this need for tangibility is how i have found myself struggling — somewhat to my surprise — to put together a draft funnel of attrition. one of the key issues is whether each step is necessary to move to the next; this issue is nicely dealt with in rick davies’s effort at a different diagram of attrition. this gets around the problem of whether, say, changes in attitude are required to precede changes in behavior.

.

but it doesn’t fully get around the challenges of a multi-year, multi-phase program and how to represent this. as noted, different stages of the program present different challenges and so it seems difficult to sum this all up as a ‘maintenance of participation.‘ this glossing over three years of maintaining participation also seems undesirable from the point of view of facilitating the articulation of reasons for dropping out along the way. things also get muddled as to whether to put outputs and outcomes between different phases of the program. again, one issue is necessity and sufficiency and can therefore be addressed as per davies’s suggestion of layered circles. another issue is that it is getting to be just a really long funnel, which is more overwhelming to look at but also a more nuanced guide to conversation and thinking.

.

we’ll see what comes out!

*addendum from 2 march 2016: the approach of considering enablers and ‘blockers’ at each step of the funnel attrition went pretty well, even though the funnel we used focused almost entirely on the programmatic phases and activities involved in maintaining/using the program over the course of three years. this means for this particular activity, we didn’t really discuss how activities are expected to translate into a series of outputs and outcomes but it still provided fodder for about two hours of discussion. one small thing that would have made the exercise easier would be adding numbers to the spaces between the ‘rungs’ of the funnel so that it was easier to have people in the same spot.