politics

Aside

“Politically Robust” Experimental Design in Democracies and a Plea For More Experience Sharing

Sometimes I re-read a paper and remember how nice a sentence or paragraph was (especially when thinking that a benevolent or benign dictator might make research so much easier, as though easy was the main goal of research).

So it is with the paper by Gary King and colleagues (2007) on “a ‘politically robust’ experimental design for public policy evaluation, with application to the mexican universal health insurance program”.

Scholars need to remember that responsive political behavior by political elites is an integral and essential feature of democratic political systems and should not be treated with disdain or as an inconvenience. Instead, the reality of democratic politics needs to be built into evaluation designs from the start — or else researchers risk their plans being doomed to an unpleasant demise. thus, although not always fully recognized, all public policy evaluations are projects in both political science and political science.

What would be nice is if researchers would share more of their experiences and lessons learned not just in robust research design (though this is critical) but also in working to (and failing to) persuade local political leaders to go along with randomization schemes and to potentially hold off any kind of scale-up until the results are in… and only if they are promising!

back (and forward) from ‘the big push forward’ – thoughts on why evidence is political and what to do about it

i spent the beginning of the week in brighton at the ‘big push forward‘ conference, on the politics of evidence (#evpolitics) which mixed the need for venting and catharsis (about the “results agenda” and “results-based management” and “impact evaluation”) with some productive conversation, though no immediate concreteness on how the evidence from the conference would itself be used.

in the meantime, i offer some of my take-aways from the conference – based on some great back-and-forths with some great folks (thanks!), below.

for me, the two most useful catchphrases were trying to get to “relevant rigor” (being relevantly rigorous and rigorously relevant) and to pay attention to both “glossy policy and dusty implementation.” lots of other turns-of-phrase and key terms were offered, not all of them – to my mind – terribly useful.

there was general agreement that evidence could be political in multiple dimensions. these included in:

  • what questions are asked (and in skepticism of whose ideas they are directed), by whom, of whom, with whom in mind (who needs to be convinced), for whom – and why
  • the way questions are asked and how evidence is collected
  • how evidence is used and shared – by whom, where and why
  • how impact is attributed – to interventions or to organizations (and whether this fuels competitiveness for funds and recognition)
  • whether the originators of the idea (those who already ‘knew’ something was working in some way deemed insufficiently rigorous) or the folks who analyze evidence receive credit for the idea

questions and design. in terms of what evidence is collected and what questions are asked, a big part of the ‘push back’ relates to what questions are asked and whether they help goverments and organizations improve their practice. this requires getting input from many stakeholders on what questions are important to ask. in addition, it requires planning for how the evidence will be used, including what will be done if results are (a) null, (b) mixed, confused or inconclusive, and (c) negative. more generally, this requires recognizing that policy-makers aren’t making decisions about ‘average’ situations but rather decisions for specific situations. as such, impact evaluation and systematic reviews need to help them figure out what evidence applies to their situation. the sooner expectations are dispelled that an impact evaluation or a systematic review will provide a clear answer on the what should be done next, the better.

my sense, which was certainly not consensus, is that to be useful and to avoid being blocked by egos, impact questions need to shift away from “does X work?” to “does X work better than Y?” and/or “how an X be made to work better?” this also highlights the importance of monitoring and feedback of information into learning and decision-making (i.e.).

two more points on results for learning and decision-making. first, faced with the assertion that ‘impact evaluation doesn’t reveal *why* something works,’ it is unsatisfactory to say something along the lines of ‘we look for heterogenous treatment effects.’ it absolutely also requires asking front-line workers and program recipients why they think something is and is not working — not as the final word on the matter but as a very important source of information. second, as has been pointed about many places (e.g.), designing a good impact evaluation requires explication of a clear “Theory of Change” (still not my favorite term but apparently one that is here to stay). further, it is important to recognize that articulating a ToC (or LogFrame or use of any similar tool) should never be one person’s all-nighter for a funding proposal. rather, the tool is useful as a way of collectively building consensus around mission and why & how a certain idea is meant to work. as such, time and money need to allocated for a ToC to be developed.

collection. as for the actual collection of data, there was a reasonable amount of conversation about whether the method is extractive or empowering, though probably not enough on how to shift towards empowerment and the fact that extractive/empowering are not synonymous with quant/qual. an issue that received less attention than it should have was that data collection needs to align with an understanding of how long a program should take to work (and funding cycles should be realigned accordingly).

use. again, the conversation of the use of evidence was not as robust as i had hoped. however, it was pointed out early on (by duncan green) that organizations that have been comissioning systematic reviews in fact have no plan to use that evidence systematically. moreover, there was a reasonable amount of skepticism around whether such evidence would actually be used to make decisions to allocate resources to specific organizations or projects (for example, to kill or radically alter ineffective programs). rather, there is a sense that much impact evaluation is actually policy-based evidence-making, used to justify decisions already taken. alternatively, though, there was concern that the more such evidence was used to make specific funding decisions, the more organization would be incentivized to make ‘sausage‘ numbers that serve no one. thus, the learning, feedback and improving aspects of data need emphasis.

empowerment in the use of data (as opposed to its collection) was not as much a part of the conversation as i would have hoped, though certainly people raised issues of how monitoring and evaluation data were fed-back to and used by front-line workers, implementers, and ‘recipients.’  a few people stressed the importance of near-automated feedback mechanisms from monitoring data to generate ‘dashboards’ or other means of accessable data display, including alternatives to written reports.

a big concern on use of evidence was ownership and transparency of data (and results), including how this leads to the duplication/multiplication of data collection. surprisingly, with regards to transparency of data and analysis, no one mentioned the recent reinhart & rogoff mess, nor anything about mechanisms for improving data accessibility (e.g.)

finally, there was a sense that data collected needs to be useful – that the pendulum has swung too far from a dearth of data about development programs and processes to an unused glut, such that the collection of evidence feels like ‘feeding the beast.’ again, this loops back to planning how data will be broadly used and useful before it is collected.

order, power, and the importance of history – hitler in india

here’s a topic i‘ve discussed in passing for the past five years and now i suppose it is time to write on it. this article just came out, covering, roughly, hitler, gandhi, and bal thackarey in indian political discussion. i am not entirely certain of article’s claims on the extent to which admiration of hitler and dissatisfaction with gandhi are part of the same conversation. or, how much of either can be attributed to thackerey. but i have certainly witnessed both the admiration and the dissatisfaction bits. i defer to maximum city on thackerey. i leave the consideration of gandhi and the birth-rupture of the indian nation-state for others.

when i first moved to chennai, i was fairly surprised to see copies of mein kampf available for sale on the streets. this sight, in turn, heightened my surprise when speaking with even well-educated indians who had never heard of judiasm (by the way, trying to use ‘you know how buddha was a hindu…’ doesn’t quite work to explain the old testament and jesus).

this ‘what are jews?’ point is disturbing for two at least two reasons.

first, india is home to several important and old jewish communities, including in kerala and in bombay (the latter were not missed by the perpetrators of the 26 November attacks in bombay). in one of my favorite books, (indian) author amitav ghosh feels a connection with an indian slave of a jewish businessman in in an antique land, placing jews in this historical context of ancient trading between india and the mediterranean. judiasm is a part of indian history and people not knowing it points to a deeper problem in awareness about ‘others’ and even ‘self.’

second, further, this point suggests large omissions in the global history taught in schools and popularly known. actually, not just global history, but indian history as well, since subhas chandra bose reached out to, and was rebuffed by, hitler to help with independence from the british. for all of hitler’s mis/use of aryan mythology, he didn’t actually seem to think all that highly of the people of the subcontinent. one might think that sort of insult would stick.

(third, the experimentation under the nazis is a key driver of research ethics today, which is yet another avenue to learn about some of the horrors in the holocaust.)

overlooking a relatively small religious group isn’t the only aberration i’ve found – also, not having heard of poland or proclaiming that south indians are the darkest-skinned people on earth or proclaiming complete ignorance (and lack of curiousity) about the beliefs of one’s muslim next-door neighbors. again, among people with master’s degrees.

to be honest, i was surprised that the students mentioned by dilip d’souza knew hitler had committed mass, systematic murder. in my experience talking with (some! only some!)  folks in india, many admire hitler and stalin (even naming children after them) in a way completely devoid of context. as far as i can tell, they see power, authority, oratory, and the ability to impose order without knowing anything of the whole ‘invading poland’ and ‘final solution’ bits. which is precisely what makes it all alarming.

it seems to be part of a craving for order and power that makes people name children after stalin, admire hitler, and proclaim that things would be better if india were more like singapore. a problem with this is that these longings seem divorced from history and context as evidenced, in part, by never having heard of ‘jews.’ it’s kind of hard to imagine what sort of instruction could teach about hitler without mentioning jews, gypsies, homosexuals, and so on. (yes, i know that sexuality and homosexuality in india are whole other cans of worms.)

of course, trying to cross the street in india – and seeing the messiness and corruption of democracy everywhere – everyone has an occasional longing for someone to impose order. the impulse for a philosopher-king, or a benevolent dictator, or someone to nicely just make decisions and get things done have been popular in the past and even now. but, as churchill said, democracy is still the best thing going given the options. sen has certainly commented on the non-need of strong-arm values to bring about development in asia and elsewhere.

democracy relies on having informed citizens – a civil civil society (e.g. here and one of my favorite diatribes, toward the end, here). has done since rome. will always do. this suggests we all have a resposibility in being informed and helping to inform.

in india, in the US, in a lot of places, we need to do better with our history, current affairs, and civics lessons. incomplete histories are dangerous things. it is not just those who don’t learn their history that are condemned to repeat it but also those who half-learn their history.

history is full of imperfect people that can teach us both how we should do things and how not to do things. we should know about both sides of past leaders. in the US ,we may largely equate hitler with evil and the fight against him as the last war we so clearly had a moral obligation to fight. anne frank is more or less required reading and we’ve seen cabaret or life is beautiful (the latter i had to watch before heading off to undergrad). this can make it all the more alarming when we hear people praise hitler or the nazi movement more generally.

some of the horrors of nazi germany may not seem so singular to those in colonies more recently gaining their independence. with good reason, and as we all should, people in india and elswhere learn and feel that the brits and americans have been plenty destructive in their own ways. this is certainly true. but hitler is a long way past imperfect and destructive. anyone looking to praise his oratory and authority needs to be fully cognizant of that.

(small addendum, 19 july 2013: http://www.huffingtonpost.com/2013/07/05/hitler-fried-chicken_n_3550351.html)

(19 feb 2014: from @urmy_shukla: http://timesofindia.indiatimes.com/home/opinion/edit-page/LEADER-ARTICLEBRHitler-as-Hero-Society-Without-a-Moral-Compass/articleshow/32382342.cms. as she notes, strangely written but gets at the odd trend, which was yet again a topic of conversation following someone pulling out a swastik-ed bandana this weekend at ragasthan.)