Checking-in during field work

It is definitely not an easy transition to go from being the researcher in-the-‘field‘ to sending someone else off to do the work that often got you into the business in the first place (and that someone else may, in turn, be relying on field managers and other similar folks to do much of the data collection and logistics).

.

Even though i check-in by phone at least once a week during field work and text/slack plenty with my direct reports, i often feel that these calls only focus on one or two urgent topics. Plus, phone calls can be frustrated and curtailed by poor connections, background noise, and other general distractions. Meanwhile, the weekly written updates i receive focus too squarely on just-the-numbers for my liking — tables of households or schools reached for first interviews or spot-checks relative to our goals.

.

This means i often miss some of the really important stuff (whether important to me, to my direct report, or both). i also fail to get a flavor of ‘the field,’ which i miss.

.

Curtailed written notes from my direct reports also mean that they are getting limited practice / professional development in writing field notes — which my background in anthropology deems imperative to doing good and reflective social science. (i should note that while i have tried to make a good practice of doing field notes myself, many (but not all) bosses and advisors never actually seemed interested in reading them, which was always a bummer. On the other hand, my parents were avid readers.)

.

Finally, by-voice check-ins can put my direct reports slightly on-the-spot if they need to provide me some tough upward feedback in terms of improving how i am supporting them and the project.

.

For all these reasons, i am going to trial a system of slightly more (semi-)structured written feedback during fieldwork times. Note that this does not replace our informal chats and texts, nor our weekly non-project check-ins (stay tuned for another post on that!). However, i am hoping that it (a) forces my direct reports to create time/space to reflect on the field week and (b) can aid with feelings of loneliness and under-appreciation while in the field.

.

Though a lot of internal crowd-sourcing from IDinsight folks (one of the great things about working at IDinsight is how seriously we take good management, our values, and our mission — so people are brimming with good ideas!), i have come up with a nascent system. The ‘system’ consists of an excel sheet with rows of questions/topics on which i hope my direct report will reflect — and then a weekly column for the direct report to fill in his/her responses and a partner column for my responses. For now, i have budgeted in my calendar 30 minutes to review and respond weekly. i hope the excel sheet will help keep things organized as well as make it easier to spot trends in the ‘mood’ in the field.

.

i will update on how it turns out but i also welcome thoughts and insights from folks in the meantime.

.

One idea from my colleagues that i have not, for the moment, decided to take up consisted of a daily written update focused on three key reflective questions to be answered at the close of the day, hopefully not adding too much time and effort to what are already usually long and tiring days:

  1. What i did today 
  2. What i plan to do tomorrow 
  3. Something i am concerned about / a challenge i am confronting

 

For now, i am going to try to focus on the weekly written update, which is a bit longer and will hopefully provide an outlet for sharing a more holistic picture of the field experience. This is still very much a beta list, so we’ll see how it goes — but i am encouraged by early and enthusiastic crowdsourcing not just from leadership at IDinsight but from Associates as well.

.

Bearing the beta-ness in mind, below please find the inaugural questions (in addition to the basic ones about survey and data quality progress) — and please make use of the comments section to suggest additional questions you have found powerful in making for good fieldwork and happy colleagues! Of course, it may be that in some weeks, the answers to these questions are simply “N/A,” but the benefit to keeping them in is, hopefully, to communicate that i am always open to hearing if something went wrong or was scary or troubling in the field.

  1. How is team morale (provide at least one piece of evidence to support that claim)?
  2. How is your morale and energy? Roughly how many hours did you work last week and how do you feel about that? What is one non-project/work thing you were able to do this week?
  3. Provide a brief update on your own and your field manager’s professional development — what is something new learned on-the-job this week and/or a gap in (tech or soft) skills that we should address? 
  4. What is one thing the field managers did exceptionally well this week? Did either face any specific challenges?
  5. Across the whole field team (from enumerators on up), what is one example of an organizational value-in-action that you have seen or heard about? 
  6. What is the most inspiring thing that happened this week (that you witnessed or heard from the field team) that relates to our mission of driving action and social impact?
  7. What was the funniest thing that happened in the field this week (that you witnessed or heard from the field team)? 
  8. What is your favorite photo you or a member of the field team took this week? Please share and explain why it was your favorite. 
  9. What was the most interesting thing that happened in the field this week (that you witnessed or heard from the field team)? 
  10. What was the most disappointing and/or scariest thing that happened in the field this week that you witnessed or heard from the field team)? (If scary, does any action need to be taken?)
  11. Overall, how was respondent and gatekeeper/stakeholder morale this week? (h/t @urmy_shukla for this idea!)
  12. What is one mistake or misstep that happened in the field this week? What changes will you make to help guard against it happening again?
  13. What is one change, if any, the field team will make before starting work next week?
  14. Any budget surprises this week?
  15. Any implementer interactions or observations about which we should be aware? How is the morale of the implementation team?
  16. Are there any things that [us supervisors] need to do or prioritize in the coming week to support field work?
Advertisements

inverted quarantines, mosquitoes & the common man in delhi

This post has been some time in the making, ever since Raul Pacheco-Vega introduced me to Andrew Szasz’s concept of an ‘inverted quarantine,’ defined further below, and fabulous Manpreet Singh and i started kicking around how the idea applied to our lives in Delhi. This week, a few events, including a desperate effort to stay awake to fend off jetlag, have conspired to help this post come together.

.

i’ll start with the idea of a quarantine, since it has such a delightful etymological origin. The phrase comes from the Italian for ’40,’ the number of days a ship needed to stay in the Venetian harbor before its booty or crew came ashore, a practice put in place during the Black Death of the later 1300s. Specifically, the ships subjected to quarantine (or forced isolation) were those returning from plague-stricken countries. The idea, as Szasz elaborates, implies the following set-up: we (Venetians) are mostly in a healthy environment, from which (potentially) diseased individuals need to be kept out. It is a collective (if enforced) action to preserve the health of the environs and, therefore, the people living in it.

.

An inverted quarantine is a response to a reversed scenario: an unhealthy environment in which individuals wish to stay healthy. Worse, these individuals have become “fatalistically resigned to it being a dangerous world” (2006). The response is a middle-class or elite response (in general) with two components intended to isolate individuals and their households/immediate environments from harm:

  • An individual response: Despite ‘the environment’ (air & water, in particular) having a generally public good quality, those constructing inverted quarantines are engaged in a response that “is individualistic in both goal and method.”
  • A consumeristic response: A sense that the way to isolate oneself and one’s family from harm requires the purchase of specialized commodities, such as bottled water.

It is the latter point, in particular, that converts a citizen (a political actor) into a consumer, who exercises a certain form of exit (to the market) rather than (political/public) voice, thus functioning as a political anesthesia.

.

This idea of building bubbles through consumer purchases has many examples in Delhi, with bottled/canister or water filters as a prime example. The air quality in Delhi has recently taken on a similar, if (deplorably) much less wide-spread, response. If you haven’t been paying attention, the air quality in Delhi is real bad (as in, the worst by measures) and air pollution is real bad in general.

.

To illustrate, here is a typical workday morning for me in Delhi, which has only been made more extreme by my recent acquisition of an air quality monitor for my house.

  1. Leave the house with my air filters (n=3) running
  2. Hope that my maid (because… India) doesn’t turn off the filters during the day
  3. Wonder how much electricity I use and therefore pollution I cause running my air filters all the time. Then promptly forget about this.
  4. Wish my maid wouldn’t leave the doors open after she makes breakfast.
  5. Chuckle about how my landlords believe that the air in their small front yard is ‘fresh’ even if the rest of the city is dirty.
  6. Get into an auto-rickshaw (open-side 3-wheeler). auto.jpg
  7. Put on my fancy, Paris-ready vog mask (mine is actually plain black — but).mask
  8. Think about how i should buy masks for my mostly faithful autowalla and then wonder if he would use them (curse my non-existant Hindi).
  9. Arrive at office, where air filters are running most of the time, except during skype calls.

.

This is not a perfect bubble but it is certainly an attempt at insulating myself nevertheless: an inverted quarantine that i have tried to construct to protect me at home, at work, and in between. i have tried to make myself part of an air-istocracy.

.

And, it is worth noting that there is relatively little political action around air quality. The government has experimented with car-reduction measures but, at least anecdotally, folks were far more interested in whether this reduced their commute time than whether it lowered the particulate matter in their air.

.

The idea of an inverted quarantine, as presented by Szasz, rests on both the individual and the consumer response. Given both my research and the current outbreaks of dengue and chikungunya and other viral fevers in Delhi, i have been thinking about mosquitoes and whether the concept applies — both whether mosquitoes constitute the sort of unsafe air/water/land of which Szasz writes and also whether the individual/household response is sufficiently consumeristic to count as an inverted quaratine.

.

To be glib, we could lump mosquitoes and the diseases they carry in with ‘bad air’ (literally the origin of ‘malaria’) and solve the first problem.

.

And the idea of a consumeristic response to mosquitoes may apply to at least some of the options available. Purchasing a bednet allows me to protect my sleeping space from most night-biting mosquitoes, though we have all had the experience that one always manages to get it. Various sprays, creams, bracelets, coils, plug-ins, and electrified tennis racquets can help to ward off mosquitoes but none of them seem to keep all of them away.

.

There are also non-consumer responses to limiting mosquitoes and therefore mosquito-borne diseases in the confines of one’s house and grounds, such as covering or draining standing water, as recommended by the government in posters, including this one from my neighborhood:

moquito

.

It is not quite clear how to classify these individual / private-good responses to larger public health problems that don’t have a strictly product-based bent within a framework of inverted quarantines — but they are certainly an important type of response in India. A similar idea can be found in the constant cleaning of private spaces but the dirtiness of public spaces (as catalogued in Maximum City, inter alia), the intense faith put in ‘home [prepared] food’ as opposed to dirty and dangerous ‘outside food’ and other ideas that mix real ideas of toxins and pathogens with older ideas of purity and pollution (i believe relating to ideas of protecting oneself from social threat, as Szasz discusses).

.

In any case, mosquitoes, unlike air quality, are getting political attention in Delhi — or at least, people are calling out the lack of political action and the over-reliance on promoting individual preventative measures in the face of an outbreak. Mosquitoes are annoying little buggers and can (visibly) get through any inverted quarantines we might construct, so perhaps this call for more public, preventative action is not surprising.

.

For my thesis research in Ghana, to better understand the context in which a new malaria treatment program was being introduced, i undertook a media content analysis of how the term ‘malaria’ was deployed and discussed in online newspapers during the relevant time period. Most of the discussion was around specific malaria donations that had come in, reporting of malaria numbers at different state health facilities, or actions that the government had or would soon take around malaria prevention and ‘environmental hygiene.’ One of the presidential candidates being covered during that time was particularly concerned about environmental cleanliness and ‘filth.’

.

But a few mentions cast malaria as a disease of common humanity, from which neither rich nor poor could make themselves perfectly safe — an actual or imagined inverted quarantine does not exist (especially against day-biting mosquitoes). It (vulnerability to malaria, mosquitoes) was used as a political symbol of issues that affected all Ghanaians. This idea of malaria and the mosquito as threatening a common humanity — the common man — has a slightly funny resonance with the current political situation in Delhi, where the ruling Aam Aadmi Party (literally, ‘the common man party’) has come under fire for insufficient public health action (rightly or wrongly) in the face of an outbreak of mosquito-borne viral fevers.

.

The idea of the non-availability of inverted quarantines in the face of the biting mosquito as a source of political action deserves more attention, as does trying to shake people out of believing that their inverted quarantines against polluted air are sufficient (or indeed, that the air is something from which one requires protection) — ideally stimulating meaningful political action. Just thinking about Delhi, i am not yet convinced that measures need to be both individual and consumeristic to act as a political anesthesia. If the goal is to explain a lack of political action, then more conceptual work is needed. For example, as long as my landlords believe their front yard is fresh/unpolluted because they keep a nice garden (and others at their club start to get worried as well), it is unlikely they will be taking any political action about air pollution. Nevertheless, the idea of an inverted quarantine and how it limits public outrage and civic response seems like a useful concept for studying urban (perhaps in particular?) responses to environmental (and pathogen?) threats, not just in the US, where Szasz focuses, but far beyond.

.

To close with a small piece from Szasz’s 2006 presentation paper for the Sustainable Consumption and Society Conference:

Inverted quarantine is a twisted and perverse sort of environmentalism. The person who engages in it clearly recognizes that there is a problem [even if misdiagnosed?], is in fact quite distressed by the problem, and intent on doing something about it. Such a person, however, is deeply pessimistic about real change, unable to imagine that things can actually improve, and therefore fatalistically resigned to it being a dangerous world.

Sounds like a lot of Dilliwallas to me.

doctor marketplace & lack of system improvement (delhi summer illness 4)

One of the first things you learn when studying health systems is how imperfect health care markets are — limited time or ability to shop around, massive information asymmetries, etc. It is interesting, then, how very marketplace-like was my experience during my most recent illness episode. It is even more interesting, i think, that this took place within the same corporate hospital system — calling into question the very benefits of such an aggregated system.

.

To briefly recap,  i had (still have, actually) symptoms consistent with chikungunya,  [~chicken-goon-yuh] which is currently breaking out in Delhi but which was not actually confirmed in my case. i managed to visit three doctors in as many days to try to figure out what was going on, which is probably the most ‘shop-around’ approach i have ever taken to a single illness episode. [The desperately curious can read the previous posts on observations from navigating the hospital and health system while being sick: here, here & here.]

.

First, i saw my regular GP. Then, with some urging, i saw a recommended GP. In the interim, i had also scheduled an appointment with a rheumatologist given (a) relevant family history and (b) that my only early symptoms were joint pain , weakness & fatigue — no fever nor rash. In this post, i just focus on the GP visits.

.

The visit to my regular GP on Tuesday led to my being tested (IgM) for chikungunya as well as dengue and malaria (as precautions than really being indicated). i learned on Wednesday morning that the IgM was negative but as my white blood cells were high, i was put on an antibiotic (for a ‘post-viral infection’) and given the obligatory paired antacid + anti-inflammatory as well as calcium for unexplained reasons. i actually don’t think an antibiotic was indicated and was a bit annoyed when i asked if anything else could be causing the joint pain and was told no, which is of course a silly statement since plenty of things cause joint pain, not all of them infectious.

.

In the interim, through the healthcare recommendation grapevine that is Delhi, i learned about another doctor who was recommending a different chikungunya test (and promoting himself as a chikungunya guru). With some urging, i followed-up there was well on Wednesday afternoon, for a (super) special clinic double-feature day.

.

The second GP i saw was in the same corporate hospital system but at a different branch, less than 5 km away. i came in with high hopes for the recommended doctor, most of which were dashed over the course of the short visit. i should note that for this visit, my (male) colleague kindly accompanied and the (male) doctor spent much of the limited attention he gave to either or us (rather than his computer screen or his phone) addressed to my boss. i have never felt so blatantly part of a capitated (pay-by-patient or ‘per head’) system as i did over the course of this week of doctor visits. (A particularly endearing moment came when i asked the doctor to explain my morning’s lab report and why the white blood cells might be elevated — and i was told to google the answer.)

.

The doctor spot-diagnosed me with chikungunya while i was still shuffling in the door, before i was able to sit down and say anything beyond ‘joint pain.’ Perhaps some patients are impressed by this sort of act. i was not. The doctor did very little looking at me and certainly never touched me. i had to really push to get out a description of the specific type of joint pain i was experiencing. Much of the time he addressed himself to a desktop computer screen, where he edited old case notes as mine, such that my print-out included inaccuracies, such as stating that i have no medical allergies (i do, including to some painkillers and anti-inflammatories; moreover, he never actually asked me this question).

.

At first the doctor tried to talk me out of getting another blood test (PCR, this time) since it was expensive (true) and since he was so certain i had chikungunya (ass).

.

He then spent at least a minute of our already poorly utilized 10-minute appointment slot to denigrate my normal GP, in a mock-humble way that acknowledged him as a junior doctor and my usual GP as a senior doctor — but also that he was much more in the trenches at his location, as opposed to her more posh and secluded (<5 km away) location. (Again, recall these are both part of the same hospital system.) He was seeing all the chikungunya cases and she wasn’t, so he knew how to spot-diagnose and which test to run.

.

He then told me that when my test came back positive, as he was sure it would (it didn’t), i should switch doctors. He also poo-pooed her having prescribed me calcium and instead prescribed a multivitamin; he also prescribed a different painkiller + antacid combination for no apparent reason. (i should note that neither doctor actually asked what i was already taking in the way of vitamins before prescribing these to me — i went ahead and bought everything i was prescribed so i could show off the detritus collected for treating a suspected virus over the course of three days.)

20160821_191032

.

Perhaps to some patients, this kind of confidence and blatant salesmanship are appealing and hearken to days of doctors-as-gods. Not to me. So, at a minimum, as a salesman, this doctor has no idea how to read a customer.

.

In fact, his actions seem the very definition of not working in a system or a sign that the system is not working. He could have, instead, said he would call my regular GP and tell her about the extent of the outbreak and about which test to run. Or report it upwards so that there could at least be systemic learning within the hospital system. But, no, he opted to promote himself. And perhaps this is what the ‘system’ incentivizes. But, if so, then what exactly is the benefit of being part of a hospital system if neither my personal records nor basic system- or city-wide learnings can be shared among doctors within and across sites?

.

To return to the initial issue of whether healthcare functions as a market, to the extent that it does, people often rely on quality indicators that may not be directly related to accurate diagnosis or perfect treatment (which are sometimes hard to assess from the patient point-of-view). So, there is cleanliness and comfort of the surroundings (the corporate chain does reasonably well on this). There is whether you feel listened to and respected as a patient (fail). Or, if your doctor isn’t particularly nice (we all secretly want to be treated by Dr. House), you should at least trust him or her but neither GP in this case did anything particularly trust-earning (and did some things that were trust-burning from my way of thinking). There is convenience (yes in terms of online scheduling but no in terms of tracking patients through the system).

.

And, one would think, there would be a benefit of aggregating learnings and best practices across the system — but this appears to not be the case. The corporates may want to think again about how they are fulfilling quality demands.

 

going viral in delhi / is diagnosis a luxury (delhi summer illness 2)

In this post, i continue to try to make research and observational hay out of my own illness in Delhi (starting here). As a quick re-cap, there was a week of severe, arthritic joint pain and weakness, which started to let us slightly right when the rash and fever kicked in. Those were mercifully short-lived but the joint pain has continued for over a month.

.

When you shuffle (quite literally since my feet weren’t keen on bending and my hips weren’t into a long stride-length anyway) into a doc’s cabin in Delhi and the first words out of your mouth are ‘joint pain’ and it is dengue and chikungunya season, these are the immediate suspects (also here for news of outbreak). One of the doctors i saw  was happy to diagnose me by sight and actually, actively encouraged me not to bother with the (pcr) bloodtest, since (he was a bit of an overconfident ass and) the test is expensive (about INR 5000 or roughly US$ 75 — definitely out of reach for a lot of patients). An earlier doctor had prescribed a cheaper test, which is more sensitive to the stage of the illness (IgM).

.

At this point, i have had the two different chikungunya tests, a dengue test, a malaria test, & a parvovirus b19 test, all of which have come back negative. If malaria had been positive, of course, it would have indicated a very different treatment course than any of the viruses. And it’s good to know if you have dengue rather than a different virus because it is possible you may need a transfusion. But at the patient-level, all the rest of these viruses have a similar ‘treatment’ protocol – fluids, rest & painkillers (plus, as it always seems in Delhi, an antacid to pair with the painkiller).

.

There perhaps isn’t much reason, then, to explore which specific virus ails you unless you, like me, find comfort in having a named illness rather than a collection of symptoms that could be named ‘a viral fever.’ (Update 9 Oct 2016: unknown viral fevers in Delhi.)

.

And anecdotally, some folks in Delhi seem comfortable just saying that they have ‘a viral’ or ‘a viral fever’  or, intriguingly, that they are going to get tested for dengue to see ‘whether it’s dengue or a viral fever.’ (See also the name of the disease and the work of many anthropologists on this kind of non-specificity vis-a-vis underlying causes.) People also don’t seem a lot of stock in the tests — colleagues and at least one of the doctors i have seen feel like i probably had/have chikungunya, blood work .

.

Which raises the question of why i have sought so many different diagnostics (yes, insurance covers it) and why a person might do so more generally. For me, i have both a desire to have a name for my diseases and also a suspicion that a virus doesn’t explain the full story of what has been an extended summer of illnesses rather than a single episode. But for a regular patient paying out-of-pocket, beyond sorting illnesses with different treatment protocols (so, parsing malaria from dengue), being able to pin a particular name to the cause of feeling unwell may not be that important — or, indeed, may be a luxury.

.

From a public health perspective, though, lack of clear diagnosis means no numbers to report upward, to understand how illness patterns are changing (including with zika looming on India’s doorstep), when there is a legitimate outbreak, etc. i say that without a complete understanding of how my test results in a private, corporate hospital (some of which were sent to a private path lab in Bombay) make it into any sort of public health statistics at all. The current numbers being reported in Delhi and the surrounds certainly seem too low relative to what doctors off-handedly say they are seeing.

.

All of this raises a few questions:

  • During an outbreak, should people satisfied with a diagnosis not based on blood-work (if it looks and walks like chikungunya, it probably is)? Is this sufficiently successful to make up for time and money saved?
  • If diagnosis (sorting between viruses, say) has more public than private benefit (since your treatment won’t change and having ‘a viral fever’ seems satisfactory), should diagnostics be subsidized? How, for whom, etc?
  • Can anyone explain to me whether and how test results from the private sector of clinics and diagnostic centers make it to official numbers? What would need to be done to improve reporting and merging of results into city- or state-wide stats?

moquito.jpg

  • Would i be more satisfied with a diagnosis of an unnamed virus in the States? Possibly — it’s certainly happened when down with non-specific ‘flu‘ symptoms that rule out the need to treat with antibiotics. But why am i more comfortable with this?

i feel like an #oddeven party pooper (reducing and working are not the same)

there are two nice, evidence-informed op-ed pieces out today on delhi’s odd-even scheme to try to reduce air pollution (here and here). the results are heartening because i didn’t have a good sense of whether a two week window of implementing a policy — to which there were many exceptions — was long enough to potentially detect a statistically significant change in meaningful measures of pollution. nor, admittedly, did i feel that i was breathing cleaner air the past two weeks. as one the articles points out, much of the anecdotal chatter has been about clearer roads, not about clearer skies.

.

since i live in delhi, am certainly affected by the air quality, and worried about my health accordingly (plume tells me every day that the situation is dire), i was pretty pleased to wake up to the headline “yes delhi, it worked.” and what has indeed happened is that good evidence (rigorously obtained, as laid out by suvojit) has been generated of a statistically significant reduction in nasty particulate matter (pm 2.5) (by 18%) during the hours the intervention was in effect.

.

this was a policy that i wanted to see work, so i am pleased that the evidence shows a reduction in the particulate matter that is driving many of my good friends out of the city (alongside many other woes). but we must be careful — whether something “worked” is more subjective than is the evidence of a reduction, which greenstone and colleagues have nicely and rapidly documented.

.

if models had predicted a 50% reduction, we wouldn’t have been so thrilled about 18%. if the government had said that every little bit counts and that even a 5% reduction would be counted by them as a success and a reason to commit to continuing the program, then indeed, 18% is quite impressive.

.

moving forward, as delhi tries to clean up its act and hopefully become a model for the rest of the country, clarifying up-front decision-points and definitions of success will be important. for the next pilots — because delhi desperately needs such measures — how will we declare, in a rigorous and defensible way, that a policy effort ‘worked’ well enough to be scaled and continued?  those of us interested in promoting the use of rigorous evidence and evaluation to inform decision-making need to be slightly cautious in our interpretations and celebrations of victory when we haven’t said up front what we’ll count as a triumph.

.

*as an addendum (31 jan 2016), it is not clear that the researchers themselves penned the title ‘yes delhi, it worked.’ for the benefit of the doubt, i am hoping that the researchers submitted something more along the lines of ‘yes delhi, odd-even reduced pollution’ and that the newspaper itself opted to change it. but the point holds that success is subjective and therefore requires a definition, preferentially ex ante.

thoughts from #evalcon on evidence uptake, capacity building

i attended a great panel today, hosted by the think take initiative and idrc and featuring representatives from three of tti’s cohort of think tanks. this is part of the broader global evaluation week (#evalcon) happening in kathmandu and focused on building bridges: use of evaluation for decision making and policy influence. the notes on evidence-uptake largely come from the session while the notes on capacity building are my own musings inspired by the event.

.

one point early-on was to contrast evidence-informed decision-making with opinion-informed decision-making. i’ve usually heard the contrast painted as faith-based decision-making and think the opinion framing was useful. it also comes in handy for one of the key takeaways from the session, which is that maybe the point (and feasible goal) isn’t to do away with opinion-based decision-making but rather to make sure that opinions are increasingly shaped by rigorous evaluative evidence. or to be more bayesian about it, we want decision-makers to continuously update their priors about different issues, drawing on evidence.

.

this leads to a second point. in focusing on policy influence, we may become too focused on influencing very specific decision-makers for very specific decisions. this may lead us to lose sight of the broader goal of (re-)shaping the opinions of a wide variety of stakeholders and decision-makers, even if not linked to the immediate policy or program under evaluation. so, again, the frame of shaping opinions and aiming for decision-maker/power-center rather than policy-specific influence may lead to altered approaches, goals, and benchmarks.

.

a third point that echoed throughout the panel is that policy influence takes time. new ideas need time to sink in and percolate before opinions are re-shaped. secretary suman prasad sharma of nepal noted that from a decision-maker point of view, evaluations are better and more digestible when they aim to build bit by bit. participants invoked a building blocks metaphor several times and contrasted it with “big bang” results. a related and familiar point about the time and timing required for evaluation to change opinions and shape decisions is that planning for the next phase of the program cycle generally begins midway through current programming. if evaluation is to inform this next stage of planning, it requires the communication of interim results — or a more thoughtful shift of the program planning cycle relative to monitoring and evaluation funding cycles in general.

.

a general point that came up repeatedly was what constitutes a good versus a bad evaluation. this leads to a key capacity-building point: we need more “capacity-building” to help decision-makers recognize credible, rigorous evidence and to mediate between conflicting findings. way too often, in my view, capacity-building ends up being about how particular methods are carried out, rather than on the central task of identifying credible methodologies and weighting the findings accordingly (or on broader principles of causal inference). that is, capacity-building among decision-makers needs to (a) understand how they currently assess credibility (on a radical premise that capacity-building exercises might generate capacity on both sides) and (b) help them become better consumers, not producers, of evidence.

.

a point that surfaced continuously about how decision-makers assess evidence was about objectivity and neutrality. ‘bad evaluations’ are biased and opinionated; ‘good evaluations’ are objective. there is probably a much larger conversation to be had about parsing objectivity from independence and engagement as well as further assessment of how decision-makers assess neutrality and how evaluators might establish and signal their objectivity. as a musing: a particular method doesn’t guarantee neutrality, which can also be violated in shaping the questions, selecting the site and sample, and so on.

.

other characteristics of ‘good evaluation’ that came out included those that don’t confuse being critical with only being negative. findings about what is working are also appreciated. ‘bad evaluation’ assigns blame and accountability to particular stakeholders without looking through a nuanced view of the context and events (internal and external) during the evaluation. ‘good evaluation’ involves setting eval objectives up front. ‘good evaluation’ also places the findings in the context of other evidence on the same topic; this literature/evidence review work, especially when it does not focus on a single methodology or discipline (and, yes, i am particularly alluding to RCT authors that tend to only cite other RCTs, at the expense of sectoral evidence and simply other methodologies), is very helpful to a decision-making audience, as is helping to make sense of conflicting findings.

..

a final set of issues related to timing and transaction costs. a clear refrain throughout the panel is the importance of the timing of sharing the findings. this means paying attention to the budget-making cycle and sharing results at just the right moment. it means seeing windows of receptivity to evidence on particular topics, reframing the evidence accordingly, and sharing it with decision-makers and the media. it probably means learning a lot more from effective lobbyists. staying in tune with policy and media cycles in a given evaluation context is hugely time consuming. a point was made and is well-taken that the transaction costs of this kind of staying-in-tune for policy influence is quite high for researchers. perhaps goals for influence by the immediate researchers and evaluators should be more modest, at least when shaping a specific decision was not the explicit purpose of the evaluation.

.

one is to communicate the findings clearly to and to do necessary capacity-building with naturally sympathetic decision-makers (say, parliamentarians or bureaucrats with an expressed interest in x issue) to become champions to keep the discussion going within decision-making bodies. to reiterate, my view is that a priority for capacity-building efforts should focus on helping decision-makers become evidence champions and good communicators of specific evaluation and research findings. this is an indirect road to influence but an important one, leveraging the credibility of decision-makers with one another. two, also indirect, is to communicate the findings clearly to and to do necessary capacity-building with the types of (advocacy? think tank?) organizations whose job is to focus on the timing of budget meetings and shifting political priorities and local events to which the evidence can be brought to bear.

.

the happy closing point was that a little bit of passion in evaluation, even while trying to remain neutral and objective, does not hurt.

Aside

john oliver on why context/setting matters

#lastweektonight, on mandatory minimums (video here, article with embedded video).

.

context is important. for instance, shouting the phrase, “i’m coming,” is fine when catching a bus but not ok when you’re already on the bus.”