this is a joint post with suvojit, continuing from 5 of 6 in the series. it is also cross-posted here.
a recent episode reminded us of why we began this series of posts, of which is this is the last. we recently saw our guiding scenario for this series play out: a donor was funding a pilot project accompanied by a rigorous evaluation, which was intended to inform further funding decisions.
in this specific episode, a group of donors discussed an on-going pilot program in Country X, part of which was evaluated using a randomized-control trial. the full results and analyses were not yet in; the preliminary results, marginally significant, suggested that there ought to be a larger pilot taking into account lessons learnt.
along with X’s government, the donors decided to scale-up. the donors secured a significant funding contribution from the Government of X — before the evaluation yielded results. indeed, securing government funding for the scale-up and a few innovations in the operational model had already given this project a sort-of superstar status, in the eyes of both the donor as well as the government. it appeared the donors in question had committed to the government that the pilot would be scaled-up before the results were in. moreover, a little inquiry revealed that the donors did not have clear benchmarks or decision-criteria going into the pilot about key impacts and magnitudes — that is, the types of evidence and results — that would inform whether to take the project forward.
there was evidence (at least it was on the way) and there was a decision but it is not clear how they were linked or how one informed the other.
reminder: scenario
we started this series of posts by admitting the limited role evidence plays in decision-making — even when an agency commissions evidence specifically to inform a decision. the above episode illustrates this, as well as the complex and, sometimes, messy way that (some) agencies, like (some) donors, approach decision-making. we have suggested that, given that resources to improve welfare are scarcer than needs, this approach to decision-making is troubling at best and irresponsible at worst. note that it is the lack of expectations and a plan for decision-making that are troublesome as the limited use of outcome and impact evidence.
in response to this type of decision-making, we have had two guiding goals in this series of posts. first, are there ways to design evaluations that will make the resultant outcomes more useable and useful (addressed here and here)? second, given all the factors that influence decisions, including evidence, can the decision-making process be made more fair and consistent across time and space?
to address the second question, we have drawn primarily on the work of Norm Daniels, to consider whether and how decisions can be made through a fair, deliberative process that, under certain conditions, can generate outcomes that a wide range of stakeholders can accept as ‘fair’.
Daniels suggests that achieving four key criteria, these “certain conditions” for fair deliberation can be met, including deliberation about which programs to scale after receiving rigorous evidence and other forms of politically relevant feedback.
closing the loop: enforceability
so far, we have reviewed three of these conditions: relevant reasons, publicity, and revisibility. in this post, we examine the final condition, enforceability (regulation or persuasive pressure).
meeting the enforceability criterion means providing mechanisms to ensure that the processes set by the other criteria are adhered to. this is, of course, easier said than done. in particular, it is unclear who should do the enforcing.*
we identify two key questions about enforcement:
- first, should enforcement be external to or strictly internal to the funding and decision-making agency?
- second, should enforcement rely on top-down or bottom-up mechanisms?
underlying these questions is a more basic, normative question: In which country should these mechanisms reside — the donor or the recipient? the difficulty of answer this question is compounded by the fact that many donors are not nation-states.
we don’t have clear answers to these questions, which themselves likely need to be subjected to a fair, deliberative process. Here, we lay out some of our own internal debates on two key questions, in hopes that they point to topics for productive conversation.
- should enforcement of agency decision making be internal or external to the agency?
this is a normative question but it links with a positive one: can we rely on donors to self-regulate when it comes to adopted decision-making criteria and transparency commitments?
internal, self-regulation is the most common model we see around us, in the form of internal commitments such as multi-year strategies, requests for funds made to the treasury, etc. in addition, most agencies have an internal but-independent ‘results’ or ‘evaluation’ cell, intended to make sure that M&E is carried out. in the case of DFID for instance, the Independent Commission for Aid Impact (ICAI) seems to have a significant impact on DFID’s policies and programming. it also empowers the British parliament to hold DFID to account over a variety of funding decisions, as well as future strategy.
outside the agency, oversight and enforcement of achieving relevancy, transparency, and revisibility could come from multiple sources. from above, it could be a multi-lateral agency/agreement or a global INGO, similar to a Publish What You Pay(?). laterally, the government in which a program is being piloted could play an enforcing role. finally, oversight and enforcement could come from below, through citizens or civic society organizations, both in donor and recipient countries. this brings us to our next question.
- should enforcement flow top-down or bottom-up?
while this question could be answered about internal agency functioning and hierarchy, we focus on the potential for external enforcement from one direction or the other. and, again, the question is a normative one but there are positive aspects related to capacity to monitor and capacity to enforce.
enforcement from ‘above’ could come through multilateral agencies or through multi- or bi-lateral agreements. one possible external mechanisms is where more than one donor come together to make a conditional funding pledge to a program – contingent on achieving pre-determined targets.however, as we infer from the opening example, it is important that such commitments should be based on a clear vision of success, not just on political imperatives or project visibility.
enforcement from below can come from citizens in donor and/or recipient countries, including through CSOs and the media. one way in which to introduce bottom-up pressure is if donors adhere to the steps we have covered in our previous posts – agreement on relevant reasons, transparency and revisibility – and thereby involve a variety of external stakeholders, including media, citizens, CSOs. these can contribute to a mechanism where there is pressure from the ground on donors in living up to their own commitments.
media are obviously important players in these times. extensive media reporting of donor commitments is a strong mechanism for informing and involving citizens – in both donor and recipient countries; media are also relevant to helping citizens understand limits and how decisions are made in face of resource constraints.
our combined gut feeling, though, is that in the current system of global aid and development, the most workable approach will probably include a mixture of formal top-down and informal bottom-up pressure. from a country-ownership point of view, we feel that recipient country decision-makers should have a (strong) role to play here (more than they seem to have currently), as well as citizens in those countries.
however, bilateral donors, will probably continue to be more accountable to their own citizens (directly and via representative legislatures) and, therefore, a key task is to consider how to bolster their capacity to ensure ‘accountability for reasonableness’ in the use of evidence and decision-making more generally. at the same time multilateral donors may have more flexibility to consider other means of enforcement, since they don’t have a narrow constituency of citizens and politicians to be answerable to. however, we worry that the prominent multilateral agencies we know are also bloated bureaucracies with unclear chains of accountability (as well as a typical sense of self-perpetuation).
while there is no clear blueprint for moving forward, we hope the above debate has gone a small step towards asking the right questions.
in sum
in this final post, we have considered how to enforce decision-making and priority-setting processes that are ideally informed by rigorous and relevant evidence but also, more importantly, in line with principles of fairness and accountability for reasonableness. these are not fully evident in the episode that opened this post.
through this series of posts, we have considered how planning for decision-making can help in the production of more useful evidence and can set up processes to make fairer decisions. for the latter, we have relied on Norm Daniel’s framework for ensuring ‘accountability for reasonableness’ in decision-making. this is, of course, only one guide to decision-making, but one that we have found useful in broaching questions of not only how decisions are made but how they should be made.
in it, Daniels proposes that deliberative processes should be based on relevant reasons and commitments to transparency and revisibility that are set ex ante to the decision-point. we have focused specifically on decision-making relating to continuing, scaling, altering, or scrapping pilot programs, particularly those for which putatively informative evidence has been commissioned.
we hope that through these posts, we have been able to make a case for designing evaluations to generate evidence useful decision-making as well as for facilitating fair, deliberative processes for decision-making that can take account of evidence generated.
at the very least, we hope that evaluators will recognize the importance of a fair process and will not stymie them in the pursuit of the perfect research design.
*in Daniels’s work, which primarily focuses on national or large private health insurance plans, the regulative role of the state is clear. in cases of global development, involving several states and agencies, governance and regulation become less clear. noting this lack of clarity in global governance is hardly a new point; however, the idea of needing to enforce the conditions of fair processes and accountability for reasonableness provides a concrete example of the problem.
Heather E. Lanthorn 