have evidence, will… um, erm? (3 of 6, relevent reasons)

this is a joint post with suvojit. it is also posted on people, spaces, deliberation.

.

in our last post, we wrote about factors – evidence and otherwise – influencing decision-making about development programmes. to do so, we have considered the premise of an agency deciding whether to continue or scale a given program after piloting it and including an accompanying evaluation commissioned explicitly to inform that decision. this is a potential ‘ideal case’ of evidence-informed decision-making. yet, the role of evidence in informing decisions is often unclear in practice.

what is clear is that transparent parameters for making decisions about how to allocate resources following a pilot may improve the legitimacy of those decisions.  we have started, and continue in this post, to explore whether decision-making deliberations can be shaped  ex ante so that, regardless of the outcome, stakeholders feel it was arrived at fairly. such pre-commitment to the process of deliberation could carve out a specific role for evidence in decision-making. clarifying the role of evidence would inform what types of questions decision-makers need answered and with what kinds of data, as we discussed here.

in considering deliberative processes, we are guided by Daniels’s, normative “accountability for reasonableness” framework (A4R). Daniels proposes four criteria to bring legitimacy to deliberations and, he argues, consequent fairness to the decision.

.

relevant reasons

this post focuses on the first A4R criterion: the “relevant reasons” that, when considered, allow “the minority [to] at least assure itself that the preference of the majority rests on the kinds of reason that even the minority must acknowledge appropriately plays a role in the deliberation.” our goal is not to assert which reasons, including evidence, provide legitimate grounds for deliberation.

rather, we outline possible categories of reasons that may, ex ante, be placed on or off the table for deliberation. we then briefly consider the role of stakeholders, arguing that their involvement is most critical for setting and vetting relevant reasons. finally, we briefly consider the implications of ‘relevant reasons’ for planning evaluations useful for decision-making.

.

efficacy and effectiveness

one set of reasons relates to the proven efficacy and safety of program’s/ policy’s content. have the materials or technologies used in a program received national regulatory approval?are they safe or appropriate for all sub-groups?

will the piloted program or portfolio of programs be judged solely on its absolute effectiveness, or if some threshold of “success” of effect size will be pre-set? in addition, will only average effects be considered or will effectiveness among certain sub-groups (e.g. historically disadvantaged) be made a separate reason?

will cost-effectiveness and affordability be considered and, if so, will benchmarks be set in advance? will decisions be taken on relative effectiveness (opportunity costs) of an intervention? if yes, relative to which other interventions (for example, other programs in the same sector or same portfolio or other programs from any sector but addressing similar outcomes)? (1)

.

feasibility and logistics

are resources for scaling (financial, material, human) allocated and ring-fenced, in case a positive decision is reached? if needed, can decision-makers mobilize resources needed for scaling? what types of information about the resources needed for scaling, and the likelihood of their being mobilized, be brought to the decision table?

given what was learned about efforts and costs in piloting the project/program/policy, is the relevant implementing agency is capable of running it? at scale? if only some state or provincial agencies are capable or have the requisite infrastructure, how should this information be used in decision making? do decision-makers want to pre-commit to an everywhere-or-nowhere decision?

will (and if so, how will) decision-makers assess genuine implementation capacity from isomorphic mimicry? will the potential to build capacity be considered in the deliberation or do decision-makers want to restrict themselves to considering what can be done with resources currently in existence?

what political considerations will be part of the deliberative process, including political realities of constituencies and lobbies in both the donor country and the country in which the pilot took place?

.

involvement of stakeholders: setting reasons and making decisions

stakeholders should have an important voice in which reasons are deemed relevant for decision-making, though the extent of “voice” is not clear (e.g.). there are two distinct roles for stakeholders in decision-making: those designated (elected, appointed, or otherwise empowered) to take certain decisions and those with stake in what decision is taken (street-level implementers (and unions thereof) and intended beneficiaries). the reasons set as relevant may gain legitimacy if they are the product of negotiations between multiple types of stakeholders.

nevertheless, we stress that it is stakeholder  involvement in setting and vetting reasons relevant to deliberation – rather than directly participating in the deliberation – that can foster fairness and legitimacy as well as feasibility and efficiency. representativeness of the decision-makers is neither necessary nor sufficient for a legitimate deliberative process that leads to a fair outcome. (2)

.

considerations in designing an evaluation for decision-making

to close, we circle back to a running theme in these posts: for decisions to be informed by evidence, evidence needs to be useful to decision-making. what, then, do the relevance of some reasons tell us about what kinds of data and evaluation questions are relevant?

we have come across numerous evaluations attached to pilot programs, designed to allow decision-makers to choose a way forward, including from among several evaluated options. has it been agreed in advance that one of those options must be chosen? Putting off such questions until an evaluation’s results are analyzed — as we have seen done in practice — sets up the unhelpful cycle of not discussing what types of evidence are desired for decision-making and, therefore, not setting up the evaluation to collect that information.

when relevant reasons for deliberation are laid out in advance, they provide guidance on what types of data need to be collected and what types of questions need to be evaluated to inform a decision. while we don’t advocate for any particular reason to be deemed “relevant,” we believe the above discussion not only informs how fair decisions on resource allocation can be taken but also highlights again that evidence – whether generated quantitatively or qualitatively – needs deep consideration in order to be deemed ‘rigorous’ from the point of view of their usability in decision-making.

in our next post, we take up the second criterion in the A4R framework: publicity and transparency in decision-making, again reflecting on what it means for the legitimacy of deliberations as well as the implications for planning for and using evidence.

.

(1) here, Brock’s work on separate spheres and indirect benefits is of interest.

(2) more on ‘democracy’ as an unsolved rationing problem can be found here.