Category Archives: research

adapted writing guide, a few thoughts on writing and pedagogy

Posted on by hlanthorn

i just spent some time summarizing and adapting someone else’s adaptation of someone else’s guide to writing (specifically for ethics and philosophy but many of the points apply more generally). i have attached it in case it is of use!

i am now assisting with this particular ethics course for the first time. it was therefore no longer surprising  but still unsettling that on almost every midterm i graded last week, i wrote “a thesis sentence would be helpful.” i‘ll note that this is a masters-level course. i thought about reintroducing the (as noted by a friend, colossal ”In-and-Out 4-by-4 Animal Style”) sandwich as teaching tool but opted against it.

i find it upsetting that in a school of public health – covering topics for which communication skills are ostensibly quite important – there so little direct emphasis on improving writing and public speaking.

first, for better or for worse, at least in my department, TAs do the vast majority of the grading. we generally don’t have the time (nor, ahem, the commensurate pay) to comment on writing style and grammar as well as content.

second, the writing resources at school are limited to one man. he does a great deal of good work but we can hardly assign all students to go to him before a paper is due, as in undergrad the professor could mandate that a paper went to the writing center before it was turned submitted  (where you were forced to read your paper out loud, which was both terrifying and extremely helpful).

third, the above point is all the more upsetting given the school’s cultivation of an international student body. that there are no writing resource that ESL (or, likely EnL) students can access when working on a specific paper is fairly upsetting.

fourth, i am of the firm opinion that one really learns how to write or present by having to comment on or grade good and bad writing or presentations. however, most times when i try to insert a mandatory “read your paper out loud to another student” or “grade a fellow student’s paper” into a curriculum, it is shot down for one reason or another. yes, it’s a pain. yes, students might go easy on each other. nevertheless, i still think is a good idea and a necessary component of taking good writing seriously. rating other students’ presentations seems to go over slightly better – but only slightly.

Writing for ethics_HEL2013

back (and forward) from ‘the big push forward’ – thoughts on why evidence is political and what to do about it

Posted on by hlanthorn

i spent the beginning of the week in brighton at the ‘big push forward‘ conference, on the politics of evidence (#evpolitics) which mixed the need for venting and catharsis (about the “results agenda” and “results-based management” and “impact evaluation”) with some productive conversation, though no immediate concreteness on how the evidence from the conference would itself be used.

in the meantime, i offer some of my take-aways from the conference – based on some great back-and-forths with some great folks (thanks!), below.

for me, the two most useful catchphrases were trying to get to “relevant rigor” (being relevantly rigorous and rigorously relevant) and to pay attention to both “glossy policy and dusty implementation.” lots of other turns-of-phrase and key terms were offered, not all of them – to my mind – terribly useful.

there was general agreement that evidence could be political in multiple dimensions. these included in:

  • what questions are asked (and in skepticism of whose ideas they are directed), by whom, of whom, with whom in mind (who needs to be convinced), for whom – and why
  • the way questions are asked and how evidence is collected
  • how evidence is used and shared – by whom, where and why
  • how impact is attributed – to interventions or to organizations (and whether this fuels competitiveness for funds and recognition)
  • whether the originators of the idea (those who already ‘knew’ something was working in some way deemed insufficiently rigorous) or the folks who analyze evidence receive credit for the idea

questions and design. in terms of what evidence is collected and what questions are asked, a big part of the ‘push back’ relates to what questions are asked and whether they help goverments and organizations improve their practice. this requires getting input from many stakeholders on what questions are important to ask. in addition, it requires planning for how the evidence will be used, including what will be done if results are (a) null, (b) mixed, confused or inconclusive, and (c) negative. more generally, this requires recognizing that policy-makers aren’t making decisions about ‘average’ situations but rather decisions for specific situations. as such, impact evaluation and systematic reviews need to help them figure out what evidence applies to their situation. the sooner expectations are dispelled that an impact evaluation or a systematic review will provide a clear answer on the what should be done next, the better.

my sense, which was certainly not consensus, is that to be useful and to avoid being blocked by egos, impact questions need to shift away from “does X work?” to “does X work better than Y?” and/or “how an X be made to work better?” this also highlights the importance of monitoring and feedback of information into learning and decision-making (i.e.).

two more points on results for learning and decision-making. first, faced with the assertion that ‘impact evaluation doesn’t reveal *why* something works,’ it is unsatisfactory to say something along the lines of ‘we look for heterogenous treatment effects.’ it absolutely also requires asking front-line workers and program recipients why they think something is and is not working — not as the final word on the matter but as a very important source of information. second, as has been pointed about many places (e.g.), designing a good impact evaluation requires explication of a clear “Theory of Change” (still not my favorite term but apparently one that is here to stay). further, it is important to recognize that articulating a ToC (or LogFrame or use of any similar tool) should never be one person’s all-nighter for a funding proposal. rather, the tool is useful as a way of collectively building consensus around mission and why & how a certain idea is meant to work. as such, time and money need to allocated for a ToC to be developed.

collection. as for the actual collection of data, there was a reasonable amount of conversation about whether the method is extractive or empowering, though probably not enough on how to shift towards empowerment and the fact that extractive/empowering are not synonymous with quant/qual. an issue that received less attention than it should have was that data collection needs to align with an understanding of how long a program should take to work (and funding cycles should be realigned accordingly).

use. again, the conversation of the use of evidence was not as robust as i had hoped. however, it was pointed out early on (by duncan green) that organizations that have been comissioning systematic reviews in fact have no plan to use that evidence systematically. moreover, there was a reasonable amount of skepticism around whether such evidence would actually be used to make decisions to allocate resources to specific organizations or projects (for example, to kill or radically alter ineffective programs). rather, there is a sense that much impact evaluation is actually policy-based evidence-making, used to justify decisions already taken. alternatively, though, there was concern that the more such evidence was used to make specific funding decisions, the more organization would be incentivized to make ‘sausage‘ numbers that serve no one. thus, the learning, feedback and improving aspects of data need emphasis.

empowerment in the use of data (as opposed to its collection) was not as much a part of the conversation as i would have hoped, though certainly people raised issues of how monitoring and evaluation data were fed-back to and used by front-line workers, implementers, and ‘recipients.’  a few people stressed the importance of near-automated feedback mechanisms from monitoring data to generate ‘dashboards’ or other means of accessable data display, including alternatives to written reports.

a big concern on use of evidence was ownership and transparency of data (and results), including how this leads to the duplication/multiplication of data collection. surprisingly, with regards to transparency of data and analysis, no one mentioned the recent reinhart & rogoff mess, nor anything about mechanisms for improving data accessibility (e.g.)

finally, there was a sense that data collected needs to be useful – that the pendulum has swung too far from a dearth of data about development programs and processes to an unused glut, such that the collection of evidence feels like ‘feeding the beast.’ again, this loops back to planning how data will be broadly used and useful before it is collected.

here’s an idea — if they are trying to tell you something, make it easy for them to do so.

Posted on by hlanthorn

there’s been a good deal of press around the unfortunately insignificant results of a major HIV prevention trial with products for women in south africa, uganda and zimbabwe. the results had little to do with efficacy of the products (a pill and a gel) but rather with the fact that most of the participating women did not use the treatments as recommended – or at all.

one potential response is to improve our behavioral interventions to support adherence to treatment regimens (and prevention regimens) and integrate these methods more directly into medication trials. adherence and persistence with medication are global problems and we are just beginning to learn – with the help of health psychology and behavioral economics – how to tackle the challenge. efforts so far include high- and low-tech solutions, though not all the promises of the former, in terms of mhealth to facilitate behavior change and adherence, have yet been borne out.

another, not mutually exclusive, response would be to actually ask the women what they would like to see and use in the way of HIV prevention – a tool which should be empowering for them. the press seems full of comments like “the women are trying to tell us something!” why does it seem that, then, for a product made for them, they have to work so hard to tell us those things? why are we not hearing more sentences that start “the women told us…” that is, why, after such a big trial, am i not hearing anything about on-going qualitative and observational follow-up efforts to learn more about what exactly didn’t work about the methods offered to women?

there’s often a lot to learn from null results (that’s science, right?) but it doesn’t just come from brainstorming what went wrong. asking helps.

i don’t suggest that people are perfectly prescient about what they need or want. often, the innovations that we can’t live without now – smartphones, for example – weren’t a need or even desire that most people could have articulated 20 years ago. as such, directly asking people what it would take to get them to engage in X desirable behavior can’t determine the research agenda. but it should certainly be part of figuring it out.

causes, explanations, & getting stuff done

Posted on by hlanthorn

@edwardcarr, i also have a confession, which is that i have a small crush on you right now for the post in which you make a confession about causality and try to disentangle causes, mechanisms, and information that can be used to understand, revamp, and scale programs. i‘d like to try to tweak the argument, especially in light of the recent excitement about the publication of null results from a bombay-based cluster-randomized trial on health and pregnancy, described here.

the tweak is to separate out two categories of information that are important and may be better gleaned from qualitative inquiry and analysis, though more adaptive/iterative and process-focused quantitative data can also play a role. i think the general mindset on what constitutes (and who has) useful and useable information is as important as the difference between quant or qual analysis.

1. how the program actually brought about an effect. this point is the focus of ed’s post, as well as levy paluck’s nice paper on qualitative methods and field experiments, which distinguishes between causal effects and causal mechanisms. mechanisms can, to some degree, be pursued and examined through ever-proliferating treatment arms in RCTs… but observation and purposive, systematic conversation are very helpful. if i read ed carr’s piece correctly, he wants more (data collected with an eye towards) explanation in order to pursue (a) a deeper understanding – beyond ‘story time’ – of what moderates and mediates the (potentially causal) relationship between X and Y within the study context, to also help us gain (b) a deeper understanding of the external validity of the findings, which could inform adaptation and replication. both are important goals for studies with any intention of scaling.

2. how the program was experienced by a range of stakeholders and how it could have been done better. this part doesn’t feature in ed’s post or levy paluck’s piece but is important. sometimes i feel like when i talk about process, everyone breaks out in log-frame hives. take a deep breath. i don’t just mean process checklists and indicators. i mean recording how things went, deviations from the study design, and seeking feedback from study participants, study facilitators, study staff, and other study stakeholders. in the bombay experiment referenced above, the team had a process evaluation officer, who consistently surveyed staff and documented  meetings with the participants. these data allowed the researchers to know, among other things, that the participating urban women “balked” at collective action but were happy to share information one-on-one — a fairly useful finding for anyone else designing a program with similar goals or in a similar population. i think the researchers could have gone slightly further in asking participants about possible explanations for the similarity of outcomes in the treatment and comparison group — but the centrality of process evaluation is clear nevertheless. in a similar vein, campos et al draw lessons from experiments that didn’t happen and propose that researchers need to “work more on delivery and better incentivize project staff.” this, too, suggests a need to better collect (and use) information on delivery process and staff perceptions of projects, which means making time (and setting aside money) to solicit this information and finding ways to incorporate it into study findings. it also means taking program design as seriously as experimental design.

in sum, explanation matters. collect data that allows for better explanation of the mechanisms underlying causal effects as well as the process by which those mechanisms were put in place. in the meantime, everyone needs to do more work on figuring out how to present these types of data in a way that is easily accessible to and valued by a variety of researchers and practitioners.

‘where the streets have no name’

Posted on by hlanthorn

that’s the title of a short article in the jan/feb 2013 atlantic– and i couldn’t think of a better one.

i have written previously about the joys of getting and giving directions in lower-income countries – specifically for research and household follow-up, although the general taxi/auto/tuk-tuk stories of trying to reach any specific location purposefully are equally fun (in hindsight).

after reading my initial post, at least one friend reminded me that people at home (in the US) aren’t always so good at directions either, too familiar with a route to think about landmarks or to remember street names, and already too accustomed to google maps & similar being able to get the job done. the atlantic article from the title, about west virginia, re-emphasizes, for one, that a lack of street names, the use of landmarks, etc, is hardly only a poor-country phenomenon – rather, that “addresses have historically been an urban commodity” and one that probably belonged to highly literate urban areas with people who moved around the city a good deal.

formalizing addresses is more important than the inconvenience of trying to find a location or getting mail delivered. it is also essential for emergency services to find you and is presumably useful for tax collection and other basic services of the state.

which brings us to the second important part of the article: west virgina relied both on 911-services and a deal with verizon to get the mapping and road-naming word underway. knowing the power and visibility of mobile companies in many low- and middle-income countries, would this not be a reasonable way  to move the task forward? of all the potential projects for m-dev (e.g. and here, h/t tom paulson), it seems to me that mapping, paying taxes, and vital registration are some of the most promising and fundamental – as well as good public-private ventures. these would be fairly top-down and possibly foucauldian projects, and may be faulted for that, but i think we need more thinking about how the state can connect with its citizens.

finally, the atlantic article also points out the fun/difficulty of coming up with that many new street names. on absurd street names presumably combined by some random generator (although the linked article points to a single woman), i think my parent’s town has to take the cake.

drive-by truckers (highway 72):

“Don’t know why they even bother putting this highway on the map
Everybody that’s ever been on it knows exactly where they’re at.”

diwali and drugs – lessons from drug sales in india

Posted on by hlanthorn

as, you know, possibly from watching The Colbert Report, it’s dwali. as stephan and wikipedia note, diwali is the festival of lights or lamps. 

being in india – or at least chennai – however, one might be hard-pressed to think that it was not the festival of sound (also, sweet pongal).  firecrackers – or ‘crackers’ – play a large role. at all hours. regardless of any noise ordinances. regardless of whether my parents thought i was under assault when talking to me on the phone. seriously, if you know a war vet that still jumps at loud noises, please avoid indian cities during diwali. dr. dischord and the awful dynne would be so pleased.

so it was (ok, and because of a hard mattress) that i went out in search of sleeping pills. up till that point, i had not needed to actually purchase drugs in india. since part of my background is in private drug sellers, i was fairly confident i would be able to get something that would get me through the exploding nights of the rest of diwali. the private drug-retail market in india is fairly infamous for being unregulated – or, ‘the free-est market’ as (many) people thought was a funny joke. imagine my surprise, then, when at drug shop after drug shop, sellers heard my request, smiled sheepishly, asked for my presciption and, when i could not produce one, refused to sell the pills to me.

 i finally found one shop at which the vendor, after looking around furtively, cut off some pills for a blister pack, stuck them in a little paper bag, and sent me off. i didn’t actually know what i had been given, so worked to reconstruct the letters visible on the back of the blister pack with my dad over the phone (no internet in chennai apt – this was 2007). i had some sort of anti-anxiety meds.

i tried asking around after that as to why my mission had been much more difficult than expected. the few non-’i don’t know’ answers i got had nothing to do with fear of state regulation of pharmaceuticals but, rather, social censure. socially, people seemed to link sleeping pills, anti-depressents, and similar drugs with attempting to commit suicide. it was the community backlash from potentially being implicated in abetting a suicide attempt to which drug vendors were responding.

besides trying a few other times to buy sleeping pills and having difficulity, i haven’t researched this issue with any particular diligence. but, if true, it may suggest ways to work on getting drug vendors to behave appropriately, even if the formal regulatory system isn’t likely to catch up any time soon.

presentation tricks & tips from the US presidential & VP debates

Posted on by hlanthorn

presidential 1: ”no, i wouldn’t have gone over the time limit if you hadn’t interrupted me to point out the limit.

VP: ”no, i can’t give you more specifics but i can repeat the same thing, slower and with hand gestures. (or try link here.)

presidential 2: don’t do this while other people are talking. the slower, non-defensive response can work quite well. also, how you listen is important.

presidential 3: actually, the third debate was pretty upsetting. maybe the daily show will pull me out of it. a few things:

  • clever lines and the now-ubiquitous ‘zingers‘ stick, even if they are untrue (and, yes, i thought obama had some good ones but, no, they weren’t fully accurate). this actually makes the lesson of romney’s leaked first debate strategy not that one shouldn’t spend time coming up with zingers, it’s just that you shouldn’t let anyone find out that you are doing so because it sounds silly if you have to practice.
  • can we stop pretending that ‘flip-flopping’ is such a horrible thing? i agree that saying different things to different audiences is bad, as is the need to change one’s opinion because you spoke too hastily the first time around. but changing one’s position or, i don’t know, updating one’s prior based on new information, should be accepted, if not rewarded.
  • can the phrase ‘you’re all over the map’ be stricken from foreign policy debates? first of all, because that should simply be a statement of fact in a foreign policy debate. and, second, because it was not a statement of fact about last night’s debate. if we were talking about a map that, say, alexander the great (actually, erosthenes) had, then, yes, perhaps the conversation on 22Oct would have appeared to be ‘all over’ it.
  • really, nothing on ‘development’ efforts? no hearts and minds and bodies and lives and livelihoods? geez, mcgovern for president.

i am not sure that means what you think it means (II): there’s a good chance the question has not been begged because no one begged it

Posted on by hlanthorn

begging the question ≠ raising the question

the folks at this site discuss the dangers and prevalence of ‘BTQ abuse.’

they note: Beg the question does not mean “to raise the question.” (e.g. “It begs the question, why is he so dumb?”) This is a common error of usage made by those who mistake the word “question” in the phrase to refer to a literal question. Sadly, the error has grown more and more common with time, such that even journalists, advertisers, and major mass media entities have fallen prey to “BTQ Abuse.”

While descriptivists and other such laissez-faire linguists are content to allow the misconception to fall into the vernacular, it cannot be denied that logic and philosophy stand to lose an important conceptual label should the meaning of BTQ become diluted to the point that we must constantly distinguish between the traditional usage and the erroneous “modern” usage. This is why we fight.

this may also be a good time to review the 2007 consideration of the logical structure of “this is why i’m hot.”

*thanks, jq!

p4p, habits, context, process

Posted on by hlanthorn

Mark Nichter and Prascilla Magrath (hereafter, M&N) have a nice new paper on pay-for-performance (P4P), stating that “understanding the processes by which P4P targets are reached demands a reorientation… towards an understanding of motivation as a component within a complex adaptive social system.” (Complex systems are so hot right now!)

the paper is worth reading in full. the authors also raise important points that i have wanted to address for some time and hopefully will do so in more depth in the future. these relate to process; motivation & context; and experiments.

1. a key question that is not always sufficiently asked or answered in health systems – or other – research is ‘how/why did X intervention not/work in Y setting?” this is an issue of both internal validity / credibility and external validity. internally, answering these questions means looking at the intervention from the perspective of multiple stakeholders to get a nuanced, full view of how or why something worked. externally, of course, the more the context and processes are understood, the easier it may be to predict when and where else the intervention may be successful.

answering these types of questions involve qualitative as well as quantitative work and looking at implementation process as well as impacts. hopefully i will have a write-up on studying implementation in the near future with @jonathan_payne [yes, Jon, I am putting it in writing as a commitment device]. the call for these types of work in the study of interventions – including but not limited to explicitly experimental approaches – include papers by Paluck, Mills, and Ssengooba, as well as the M&N paper discussed here.

i want to quickly make amends for the past times i have denigrated process indicators. it’s true that process indicators can become as meaningless as they are made into check boxes of routines (such as number of posters hung or meetings held). also, process indicators without impact indicators don’t get us as far as we need to go in saying whether something worked or should be done again. but, recording and evaluating process is necessary, if not sufficient, in saying how something worked and whether it could work again (and where).

2. M & N draw on Bordieu to provide a frame for studying the implementation and impacts of P4P – but the point is more widely applicable. M & N draw on the concept of ‘habits’ – attitudes, dispositions, actions – of individuals and social groups to help explain how interventions are received differently in different places. this has interesting parallels with Stein‘s work on ‘habits,’ for which he draws on the work of Velben; these parallels and lessons deserve more exploration than i give them here.

both roughly aim at the idea of shaping interventions to a given contextual reality, including political, economic, and administrative structures but also social processes and group, sub-group, and individual dispositions and ‘habits of thought.’

both also suggest the need to consider how an intervention’s positive & negative effects will ripple out in both time and space, as well as how positive effects can be sustained. Stein highlights that “relations among institutional constructs, habits and the transformation of behavior are at the core of development.” of course, M & N also point out that changing behavior is not everything. behavior relates to what actors ’will do’ but what they ‘can do’ is may still be constrained by a lack of material, human, and time resources.

M & N also draw on Bourdieu’s multiple forms of capital to consider the ways that interventions may change dispositions and behavior; these include economic but also cultural, social, and symbolic capital. the importance of non-economic forms of capital in changing motivations and incentivizing behavior are increasingly recognized, including the many talks of Rory Sutherland and Ashraf’s recent paper on pro-social benefits. the way an intervention will change the distribution of all these types of capital should be considered in the design and the process captured over the course of implementation.

the ways that previous interventions have altered the distribution of these forms of capital is also an important consideration; tabula rasa non existunt.

3. finally, for quite some time (with a little e-input from Owen Barder – thanks!), i have been working up to saying all social interventions – explicitly experimental or not – are experiments. they change the context in ways that are not adequately considered. this includes the ethical ramifications of implementing an intervention and of stopping that  intervention. at the very least, the presence of an experiment/intervention changes changes expectations. while i’ve been tiptoeing up to this point for months, M & N go right ahead and say, “short-term interventions can have long-term impacts on expectations.” an experiment or intervention might be billed as discrete, pilot, or otherwise, but if it is taking place outside a controlled laboratory setting, it is bringing about some change, from anchors and reference prices to much larger shifts in thinking on ‘how things work.’  experiments have social and political ramifications beyond the intended effects, at levels above the individuals directly involved.

to this end, M&N and others promote a ‘cyclical formative reformative research approach’ as a way of moving forwards with (health) systems research and i strongly back this idea as a way of experimenting and promoting development more generally. this sort of long-term research agenda does not always fit with the present structure of grants & funding but hopefully the latter will begin to change.