Addressing courtesy bias

This is a post that i recently penned for IDinsight’s internal blog but i thought was worth pushing out into the interwebs, in hopes that people will weigh in with tactics they have tried!

Please share in the comments your experiences with courtesy bias and the tactics you have used (seemingly successful and unsuccessful) — including the ones here, which are not fool-proof.

Continue reading

Advertisements

Daily(ish) field debriefs for qualitative work

While i miss being in ‘the field’ and conducting interviews myself, one way that i remain connected to the research is by conducting frequent debriefs during the data collection (and early analysis phase). Debriefing isn’t just self-serving; it can improve the quality of qualitative data collected and the subsequent analysis.

.

After a particularly insightful debrief this morning, i thought i’d quickly put together some thoughts about the process and why it is so important if you are conducting qualitative work.

.

There are a few hallmarks of a qualitative approach research that make it important to check-in regularly during the data collection (interviews, observations, focus groups) process in order to strengthen and make more credible the information being produced.

.

  1. The data collector and data analyst are often the same person or small set of people (which generally implies a higher skill level for a qual data collector than the quant counterpart).
  2. A qualitative approach to research is an iterative endeavor — this is a key distinguishing feature from a lot of large-n, more quantitative work. In quantitative work, we usually take as fixed all the items will include in a questionnaire, which serves to enhance rigor and decrease bias.In qualitative work, we have a bit more flexibility and pursue rigor a little differently. This allows us to incorporate learnings from one day’s interview work into the next day’s interviews (or observations, or focus groups), as we continue to update our understanding. For example, we can incorporate into the interview guide a new probe that worked particularly well at eliciting a rich response; or, a new line of thinking might emerge that we want to explore with future respondents; or, we may notice a potential pattern emerging that we want to pressure-test the next day.

.

There are some important implications of these features.

  • First, it is very easy to get stuck in the weeds if you are the data collector, focused on scheduling the next interview and generally being tired after so much human interaction, rather than taking time to reflect on what you have learned. It is not good to be mired in executing-but-not-reflecting-and-and-hypothesizing-and-adapting.
  • Second, you need to have time built in to your schedule and budget to reflect and update your approach. This time can come between interviews during the day, as well as allotting time between days of interviewing. This protected time can be used for transcription and expanding field notes, for reviewing these notes and transcripts, and to begin early analysis (such as memoing and early coding). This helps to prevent many cases of “we should have asked that!” once we get back from ‘the field.’ By doing this as we go, it is easier to return for follow-up interviewing or at least ask the rest of our participants some additional questions. It also helps us to learn more from each interview.

.

To facilitate this reflection without getting stuck in your own head, it can be extremely valuable to have a debrief with someone else about what you have been learning and experiencing in the field, to start to make sense and meaning of what you have been hearing. These debriefs — sometimes called peer debriefs* — can take the form of written answers shared with the larger team or can take place on phone calls or face-to-face with another team member (who did or did not participate in the interview or observation). Technology and timezones will determine what mode is most sensible; it may change over the course of a project. (Note that debriefs are also a great way to make sure that team members not in ‘the field’ still feel involved in the project.)

.

These debriefs are not substitutes for early analysis or for another member of the team reviewing transcripts and field notes to provide feedback on: points that have been over- or under-emphasized, points where responses seem vague and more probing may have been required, or points the interviewer seems to have made unnecessary or unhelpful assumptions.

.

Below are the set of questions i am currently asking for reflection:

  • What was an interesting point that arose in the interviews today? Does this suggest an additional set of questions to ask in future interviews?
  • What surprised you most about the interviews today?
  • Did anything particularly challenging or difficult or unpleasant come up during the interviews today? Should we think of ways to mitigate these challenges in the future?
  • Are there any questions that don’t seem to be yielding useful information? Should we update or remove these questions?
  • Do any patterns stand out to you in responses (early coding and categorization)? Does this suggest any hypotheses that we may want to pressure test in future interviews?

.

*Some people argue that peer debriefs should not be conducted by a member of the research team, in pursuit of impartiality. Ideally, both someone from the research team and a neutral person outside the team can be reviewing data as it is coming in and asking questions to strengthen the research process.

A brief history of jerrycans

rally Thanks to dissertation-writing procrastination a few years ago, I watched all eight series of Foyle’s War in a somewhat absurdly short amount of time. This, perhaps embarrassingly, is how I became familiar with WWII (and probably WWI) slang, such as calling Americans ‘Tommies’ and Germans ‘Jerries’ (the reasons behind German nickname are a point of speculation). Perhaps this was not the best way to learn about military history. But.

.

This led me to take a second look at jerrycans a new way on my most recent trip to sub-Saharan Africa (specifically in Lusaka). Might these containers be named after Germans? (Short answer: yes.) Might they be engineering marvels with deep historical importance that i had never appreciated before? (Short answer: also yes.)

.

Perhaps ‘jerrycan’ is a word lots of you folks were already using for water containers and had already thought deeply about — but it is a word with which i became familiar only in Ghana, where the colorful containers are ubiquitous (this picture is actually from my recent trip to Zambia; the yellow can is the one i am talking about).

IMG_20170915_103650

Mtendere, Lusaka 2017

i associate the term with being used in Africa, being plastic, and with carrying water (as here). But the plastic containers are premised on a (revolutionary) German, pressed-steel, fuel-carrying model from 1937, nicknamed jerrycans by Allied troops. The originals were designed under secretive conditions to carry 20 liters, not require a funnel to fill or a wrench to unscrew, to be stackable, and — strengthened and made flexible by their X-shaped indentation — be durable beyond a single use. Evidently, Hitler already had thousands of these containers (Wehrmachtskanister) stockpiled by 1939.

.

While i think of jerrycans as being carried on the head (here’s a picture of a woman carrying 20 (!) this way), the design was intended to allow a soldier to carry two full or four empty cans – evidently quite an improvement on existing models (read here on “epic ergonomic failure”). The plastic version made a 1970 debut thanks to a Finnish engineer. They became commercially available shortly thereafter. In places like Africa (yes, broadly), they have provided a useful alternative to clay or metal pots for collecting and storing water and they are often a mainstay of NATO and UN efforts.

.

Apparently (note that much of this comes from Wikipedia) the jerrycan is quite an engineering marvel and its history is exciting because of the extreme importance of fuel and efficient (literally non-leaky) fuel distribution to the war effort and the reverse engineering of the design by Allied forces.

.

In one iteration of the story (relayed on Wikipedia and, seriously, on jerrycan.com), the model was effectively smuggled out of Germany by American engineer Paul Pleiss, who drove from Germany to India in 1939 — with a German colleague and three cans in tow (it is not clear whether part of the goal of this trip was to smuggle the cans out or if that was incidental; it seems likely that they just needed water and, along the way, Pleiss realized the cans were kind of neat). The German revealed the specifications for the cans before being recalled to Germany by Göring. Pleiss managed to get the cans back to the US from Kolkata and sent one to the US Army at Camp Holabird (to do this without alerting anyone to what interested him, he had the whole car shipped back via Turkey, cans included). The US eventually redesigned the German model to suit its needs, poorly at first and better later in the war (this blog post appears to be written by a WWII vet and discusses how he and a colleague got the US Navy interested in the design, in 1942).

.

Meanwhile, the British apparently first saw the German design in April 1940 in Norway and realized it was superior to their thin tin or mild-steel ‘flimsies,’ which leaked and often only endured a single use (at least as a carrying device; much is made online of their being turned into Benghazi Burners).

.

In 1940, Pleiss sent one of his cans to London for them to copy and manufacture. Still, the flimsies were very much part of the British arsenal as the war opened in the North African theater (in June 1940) and the British armies there and in Europe did what they could to get their hands on the preferable German models. It seems that the continued production of flimsies for British troops prior to 1942, even while the troops were witnessing the superiority of the jerrycans, was a source of bitterness and demoralization. After this, the British ramped up production and were the main suppliers for Allied troops. Evidently, President Roosevelt (in November 1944) attributed to the cans the speed with which Allied forces were able to move across France following the invasion at Normandy.

.

All of which is to say, if you find yourself looking at a plastic water container and thinking that it is just a boring old water container, you are quite wrong.

Figuring out how to manage direct reports

Before starting with IDinsight, i had only limited managerial experience — mostly managing field managers and survey teams. i certainly had not given much thought to management or how to do it in a meaningful way, beyond a general sense of wanting to get good work out of my team as well as keep them happy. Nor had i really had what i considered to be a stellar experience of being managed. i had a great academic committee, for example, but i wouldn’t hold that up as the best way to manage people. And i have had an array of other less-than-fun managers, including micro-managers and credit-stealers, in organizations that didn’t necessarily put any value on spending time and effort on becoming a better manager. 

.

i found it quite daunting, then, to find myself with one — and later two — direct reports in an organization that takes management, feedback, and professional development quite seriously (one of the things that makes IDinsight a fab place to work). (Note that the folks i manage are a few years out of undergrad, sometimes with a 1-year masters degree.) 

.

i certainly talked to my direct reports every day (when i first started, there were only six of us sitting in a very open office, so it would have been quite hard to not talk to my direct report!) but i still felt it wasn’t enough: it neither (intentionally) set him up well to reflect on his work and his growth, nor set me up well to do performance reviews and other necessary things. 

.

Early in my tenure with IDinsight, i had a very positive experience of working through the book Managing to Change the World with a colleague with more management experience inside and outside IDinsight. One of the big takeaways (for me) from the book and our conversations was the usefulness of setting aside explicit time/space for non-project check-ins — no matter how much you already talk to discuss project work or chat informally. The book, my colleague, and other colleagues helped provide a lot of great ideas that i crowdsourced…

.

…and then i found myself totally overwhelmed by the prospect of trying to remember all those good questions. 

.

So, as has become my management habit/style over the past 1.5 years, i created a spreadsheet, with rows for topics on which i want to check-in regularly and columns for each week. This functions as a semi-structured interview guide as well as a recording device for me, so that i can look at previous notes and patterns as well as have notes when i need to fill out performance reviews (i also find completing performance reviews very daunting and still struggle with recording peak areas and challenge areas on a daily or weekly basis!).

.

At first, i worried it would seem too rigid, if i had my laptop out during our chats and i was obviously reminding myself of the questions i wanted to ask and typing down notes. But overall, this seems not to have been a major problem. And, actually, have heard from my direct reports that they like knowing some of the questions i’ll ask in advance, so that they can reflect on them before we meet.

.

The system is certainly not perfect and is continuously updated. But one good sign (i think) is that my direct reports have started using a similar system with their direct reports (Field Managers, who are awesomely permanent staff at IDinsight) and use some of the questions that i generated and that they particularly like.

.

Although i created the initial set of questions, over the weeks and months, some have been dropped and others have been suggested and added (e.g.“Heather, will you please check in with me about X each week to help hold me more accountable to it?”).

.

One thing i have definitely learned over the past 1.5 years is that attempting to be a good manager takes time (and effort). These non-project check-ins, which we do weekly on Monday mornings or early afternoons (i strive for ‘management Mondays’), take about an hour and usually result in about 15 – 30 minutes of follow-up work based on the conversation (sending a paper i promised during our check-in, delivering a shout-out to someone i’ve discovered helped on our project in some unseen way, etc). i’ve had to learn to budget in that follow-up time so that the rest of my schedule isn’t thrown off.  

.

Below are the questions i currently aim to ask every week, though we don’t always make it through all of them (which i think is ok); as noted above, it should be considered a semi-structured interview guide to help have a conversation on important topics that can easily be elided in favor of easier small talk or in favor of focusing strictly on the project work. As much as possible, my goal is to help raise important issues — but give my direct reports a chance to come with ideas of how to tackle those issues. Theoretically, this is good professional development for them and, ideally and honestly, a little less work for me! 

.

Hope this post is useful to some other folks who are attempting to figure out how to be a good manager, despite not necessarily having had one before or not having worked in an organization that took it seriously before. The questions below reflect IDinsight’s focus on professional development, organizational contribution, mission-focus, and values — as well as things stolen from the aforementioned management book.

.

Feedback is, as always, welcome — particularly if you try out this list of questions and want to share your experience or, especially, if you have a go-to check-in question that works well for you!

  • Are you keeping healthy/well?
  • Are you happy and motivated? Really? Why/not?
  • What is the best thing that happened over the weekend?
  • What are two things that will make this a successful week for you?
  • What’s the toughest thing coming up this week? Anything that would make you feel more confident about tackling it?
  • Is there one tweak we could make that would make this a more manageable week?
  • Did it feel like you had roughly the right amount of work / working hours last week? Does it feel like you are working at a sustainable pace? On sustainable topics? (What would be sustainable to you?)
    • Reflecting back on last week, how many nights, if any, did you lose sleep over work?
  • Last week you were worried about [X] as the toughest thing coming up. How did it go? What worked and what could have been improved?
  • What is one new thing you learned or did last week related to professional development? (Here sometimes i probe for lessons related to written and oral communication, management, organizational development — folks have a gut tendency to equate professional development only with coding, even though technical proficiency is only one of our six performance review categories.)
  • What is one new thing you learned or did last week related to technical professional development?
  • Did this week throw up any gaps that you feel in your technical and professional development? Any thoughts on how we might address these?
  • What is one professional development task or goal you feel you did not achieve last week? How might you be able to fit it in in the following week?
  • Did you try any new strategies to manage your time or engage in deep work last week? Were these successful? Why or why not? What might you try this week?
  • What was the most interesting or helpful piece of feedback or advice you received last week?
  • Which aspect of your work gave you the most joy or fulfillment last week?
  • Is there that I could have done this past week to better support or coach you? What could I have done differently?
  • Do you feel like there is anything else you could be contributing to organizational growth and development that you are currently not (but would be interested in doing)?
  • Do you feel like we influenced any decisions, actions & social impact last week? What worked or didn’t in that process?

Checking-in during field work

It is definitely not an easy transition to go from being the researcher in-the-‘field‘ to sending someone else off to do the work that often got you into the business in the first place (and that someone else may, in turn, be relying on field managers and other similar folks to do much of the data collection and logistics).

.

Even though i check-in by phone at least once a week during field work and text/slack plenty with my direct reports, i often feel that these calls only focus on one or two urgent topics. Plus, phone calls can be frustrated and curtailed by poor connections, background noise, and other general distractions. Meanwhile, the weekly written updates i receive focus too squarely on just-the-numbers for my liking — tables of households or schools reached for first interviews or spot-checks relative to our goals.

.

This means i often miss some of the really important stuff (whether important to me, to my direct report, or both). i also fail to get a flavor of ‘the field,’ which i miss.

.

Curtailed written notes from my direct reports also mean that they are getting limited practice / professional development in writing field notes — which my background in anthropology deems imperative to doing good and reflective social science. (i should note that while i have tried to make a good practice of doing field notes myself, many (but not all) bosses and advisors never actually seemed interested in reading them, which was always a bummer. On the other hand, my parents were avid readers.)

.

Finally, by-voice check-ins can put my direct reports slightly on-the-spot if they need to provide me some tough upward feedback in terms of improving how i am supporting them and the project.

.

For all these reasons, i am going to trial a system of slightly more (semi-)structured written feedback during fieldwork times. Note that this does not replace our informal chats and texts, nor our weekly non-project check-ins (stay tuned for another post on that!). However, i am hoping that it (a) forces my direct reports to create time/space to reflect on the field week and (b) can aid with feelings of loneliness and under-appreciation while in the field.

.

Though a lot of internal crowd-sourcing from IDinsight folks (one of the great things about working at IDinsight is how seriously we take good management, our values, and our mission — so people are brimming with good ideas!), i have come up with a nascent system. The ‘system’ consists of an excel sheet with rows of questions/topics on which i hope my direct report will reflect — and then a weekly column for the direct report to fill in his/her responses and a partner column for my responses. For now, i have budgeted in my calendar 30 minutes to review and respond weekly. i hope the excel sheet will help keep things organized as well as make it easier to spot trends in the ‘mood’ in the field.

.

i will update on how it turns out but i also welcome thoughts and insights from folks in the meantime.

.

One idea from my colleagues that i have not, for the moment, decided to take up consisted of a daily written update focused on three key reflective questions to be answered at the close of the day, hopefully not adding too much time and effort to what are already usually long and tiring days:

  1. What i did today 
  2. What i plan to do tomorrow 
  3. Something i am concerned about / a challenge i am confronting

 

For now, i am going to try to focus on the weekly written update, which is a bit longer and will hopefully provide an outlet for sharing a more holistic picture of the field experience. This is still very much a beta list, so we’ll see how it goes — but i am encouraged by early and enthusiastic crowdsourcing not just from leadership at IDinsight but from Associates as well.

.

Bearing the beta-ness in mind, below please find the inaugural questions (in addition to the basic ones about survey and data quality progress) — and please make use of the comments section to suggest additional questions you have found powerful in making for good fieldwork and happy colleagues! Of course, it may be that in some weeks, the answers to these questions are simply “N/A,” but the benefit to keeping them in is, hopefully, to communicate that i am always open to hearing if something went wrong or was scary or troubling in the field.

  1. How is team morale (provide at least one piece of evidence to support that claim)?
  2. How is your morale and energy? Roughly how many hours did you work last week and how do you feel about that? What is one non-project/work thing you were able to do this week?
  3. Provide a brief update on your own and your field manager’s professional development — what is something new learned on-the-job this week and/or a gap in (tech or soft) skills that we should address? 
  4. What is one thing the field managers did exceptionally well this week? Did either face any specific challenges?
  5. Across the whole field team (from enumerators on up), what is one example of an organizational value-in-action that you have seen or heard about? 
  6. What is the most inspiring thing that happened this week (that you witnessed or heard from the field team) that relates to our mission of driving action and social impact?
  7. What was the funniest thing that happened in the field this week (that you witnessed or heard from the field team)? 
  8. What is your favorite photo you or a member of the field team took this week? Please share and explain why it was your favorite. 
  9. What was the most interesting thing that happened in the field this week (that you witnessed or heard from the field team)? 
  10. What was the most disappointing and/or scariest thing that happened in the field this week that you witnessed or heard from the field team)? (If scary, does any action need to be taken?)
  11. Overall, how was respondent and gatekeeper/stakeholder morale this week? (h/t @urmy_shukla for this idea!)
  12. What is one mistake or misstep that happened in the field this week? What changes will you make to help guard against it happening again?
  13. What is one change, if any, the field team will make before starting work next week?
  14. Any budget surprises this week?
  15. Any implementer interactions or observations about which we should be aware? How is the morale of the implementation team?
  16. Are there any things that [us supervisors] need to do or prioritize in the coming week to support field work?

A small research agenda on political understanding in the U.S

From recent reading, like Strangers in their own land and The politics of resentment, three key themes keeps coming up, which suggest to me a possible research and/or journalistic agenda.

  1. An over-estimation of how many people work for the government in different forms and at different levels, how competently and efficiently they do their job, and what benefits they receive in exchange.
  2. A (continued) misconstrual of who receives different types of benefits from the government and whether they are ‘deserving,’ where dessert is tied in directly with conceptions of ‘hard work’ and whether or not they perform it.* Overall, showcasing the normalcy and neighborliness of such folks.**
  3. A misunderstanding or willful forgetting of the ‘good’ things that the government does in one’s everyday life (and how better government could do it better). This could include, for example, a sense that ‘folks like me’ don’t benefit from the ACA and other government efforts big and small — which may suggest ways forward on mitigating such views.

.

*Note that in Politics of resentment, at least, it becomes clear that hard work is often equated with physical/manual work. This is another avenue for possible exploration.

*This is, of course, linked to (a need to contrast) media portrayals of folks that you don’t necessarily interact with in-person on a day-to-day basis. See, e.g., Aziz Ansari’s Muslims eating nachos pitch. Also, this except from Strangers in their own land:

The stock image of the early 20th century, the ‘Negro’ minstrel, a rural simpleton, the journalist Barbara Ehrenreich notes, has now been upgraded, whitened, and continued in such television programs as Duck Dynasty and Here Comes Honey Boo Boo. ‘Working class whites are now regularly portrayed as moronic, while blacks are often hyperarticulate, street smart… and rich’ (p 156).

Aside

Packard on History of Global Health: devastating first chapter (we need to do better)

While i intend to write more about Packard’s new book (delightfully if uncomfortably subtitled, interventions into the lives of others) once i am through with it, a paragraph in the opening chapter seemed both so important and accurate as to merit sharing immediately — particularly given the lessons it may hold for the Universal Health Coverage (e.g.) movement. It is not that what Packard has to say here is necessarily new but rather that he sums it up in a neat, indicting list of trends, on which we would all do well to reflect:

.

There have been remarkable continuities in how health interventions have been conceived and implemented over the past century… [that] have worked against the development of effective basic-health systems to address social determinants of health:

  1. Health interventions have been largely developed outside the countries where the health problems exist, with few attempts to seriously incorporate local perspectives or community participation in the planning process…

  2. Health planning has privileged approaches based on the application of biomedical technologies that prevent or eliminate health problems one at a time.

  3. Little attention has been given to supporting the development of basic health services.

  4. The planning of health interventions has often occurred in a crisis environment, in which there was an imperative to act fast. This mindset has privileged interventions that are simple, easy to implement, and have potential to quickly make a significant impact…

  5. Global health interventions have been empowered by faith in the superioity of Western medical knowledge and technology…

  6. Health has been linked to social and economic development — but this connection has focused primarily on how improvements in health can stimulate economic development, while ignoring the impact that social and economic developments can have on health. The social determinants of health have received little attention.

.

Packard notes that these trends have faltered a few times, such as with the rise in interest in learning about the “social and economic causes of ill health” in the 1920s and 30s and in the Alma Ata / health for all movement at the end of the 1970s. We seem to think of ourselves as standing at a new trend-breaking moment. Hopefully we can do better.