This is an effort to develop a core set of questions for someone new to impact evaluation to start to learn it from the empirical literature. Specifically, this is for someone with limited exposure to these methods.
Working list:
- What did this paper tell you about the context of the study? What else do you wish you knew?
- What did this paper tell you about the intervention under study? What is/not convincing to you about the ‘theory of change?’ (They might not use that precise term but you can look up what it means!) What else do you wish you knew? I find it can be useful to draw a diagram.
- What was the key outcome(s) explored in the study? Did you think the authors overlook anything important?
- What is the key research question(s) under investigation? Specifically, what is the causal claim the authors hope to make?
- What is the key assumption(s) the authors made to allow this analysis to answer the research question posed?
- What did you understand about the methods that they used to collect data? What questions do you have?
- What did you understand about the methods they used to analyze and make a causal claim from the data? What questions do you have?
- (From what you understand) Do you believe the results? What are all the ways this paper could be wrong (and what do the authors do to reassure you it is not)?
- How did the authors visualize the results (if at all)? What would you have done differently to make it clearer?
- What do you think about the policy conclusions or recommendations they make? Do they seem like a tight fit with the results? What did they write that went ‘too far?’ What, if anything, did they miss?
- You might also see if any blog posts or other items have been written about this paper to provide additional detail
- This might be helpful as you think about approaching your reading: http://macartan.nyc/teaching/how-to-read/
Heather E. Lanthorn 