Qualitative approaches: contribution analysis

Contribution analysis is a comprehensive qualitative approach that integrates both qualitative and quantitative data to examine a programme's cause and effect relationships, enabling conclusions about its impact on outcomes. It is suitable for evaluations where experimental designs are not feasible.

Page contents

Woman farmer controls drone sprayer with a tablet

Basics

In a nutshell

A contribution analysis is used to assess causal questions and infer causality in programme evaluations. It provides a structured, step-by-step approach that helps policymakers arrive at conclusions about the contribution the respective programme has made (or is currently making) to particular outcomes. A key question in the assessment of programmes is that of attribution – to what extent are observed results due to the programme’s activities, rather than other factors, has the programme made a difference and does it have added value?

The essential value of a contribution analysis is that it is designed to reduce uncertainty about the contribution of an intervention by increasing the understanding of why observed results occurred (or did not) and the roles played by the intervention and other internal and external factors. When it is not practical to design an experiment to assess performance, a contribution analysis can provide a credible assessment of causes and effects.

This approach is particularly useful in situations where the programme is not experimental i.e. where a programme has been funded based on a relatively clearly articulated theory of change and there is little or no scope for varying how it is implemented. A contribution analysis helps confirm or revise the theory of change but is not designed to uncover implicit theories of change. Verifying the theory of change that the programme is based on, and paying attention to other factors that may influence outcomes, provides reasonable evidence about the contribution being made by a programme.

The report from a contribution analysis is not definitive proof but rather provides evidence and a line of reasoning to draw a plausible conclusion that, within a certain level of confidence, a programme has made an important contribution to documented results.

The process involves verifying the theory of change behind a programme, ensuring a programme’s activities were implemented as planned, using evidence to confirm the expected chain of results and assessing other influencing factors while recognising their relative contributions.

This method is valuable for managers and evaluators when experimental designs are impractical, offering a credible assessment of cause and effect by verifying the theory of change and considering other influencing internal or external factors.

Pros and cons

Advantages	Disadvantages
Provides a clear, systematic process for exploring cause and effect relationships. Can be applied in various contexts, especially where experimental designs are not feasible. Allows for continuous refinement and the strengthening of the contribution story as new evidence is being collected. Incorporates multiple sources of evidence, including qualitative and quantitative data.	Requires significant time and effort to gather and analyse evidence iteratively. The quality and reliability of the analysis heavily depends on the availability and quality of existing data. Developing a detailed theory of change and assembling a credible contribution story can be complex and challenging. The process may involve subjective judgments, particularly in interpreting qualitative evidence and constructing the contribution story. While it addresses attribution challenges, it may not fully eliminate uncertainties, especially in complex programmes with many influencing factors.

When to use?

In the context of the CAP Strategic Plan, contribution analysis can be used in assessing programmes developed based on a theory of change. For example, Agricultural Knowledge and Innovation Systems (AKIS) and Local Development Strategies (LDS). It can also be used in other cases where the direct attribution of outcomes to specific interventions is challenging due to multiple influencing factors and complex programme structures.

In the context of an AKIS assessment, for example, the contribution analysis can depict or disentangle multiple factors that have interacted to achieve results. It can be used to construct different impact pathways. It is useful for providing evidence and a line of reasoning from which one can draw conclusions about the contribution of AKIS interventions to expected results.

When applied in a participatory way, a contribution analysis is valuable in ensuring the significance and reliability of the data collected. Moreover, it helps stakeholders systematically evaluate which outcomes have been achieved and by whom, and manages implications and adjustments in strategies needed to achieve desired outcomes.

Preconditions

A (relatively) clearly articulated theory of change to be used as a basis.
Sufficient data must be available and ongoingly collected to support the theory of change.
A thorough understanding of a programme’s context, including external factors that might influence outcomes, is necessary to accurately assess its contribution to observed changes.

Step-by-step

A contribution analysis has six iterative steps. Each builds the contribution story and addresses identified weaknesses from the previous stage. If appropriate, many of the steps can be undertaken in a participatory mode.

Step 1 – Set out the attribution problem to be addressed

Acknowledge the attribution problem.
Determine the specific cause-effect question to be addressed.
Determine the level of confidence required.
Explore the type of contribution expected.
Determine other key influencing factors.
Assess the plausibility of the expected contribution in relation to the size of the programme.

Step 2 – Develop the theory of change and the risks to it

Determine the level of detail.
Determine the expected contribution of the programme.
List the assumptions underlying the theory of change.
Include consideration of other factors that may influence outcomes.
Determine how much the theory of change is contested.

Step 3 – Gather existing evidence on the theory of change

Assess the logic of the links in the theory of change.
Gather evidence on results and activities, assumptions and other influencing factors.
- Create an initial narrative explaining how programme activities contribute to observed outcomes.

Step 4 – Assemble and assess the contribution story, and challenges to it

Analyse the theory of change with collected evidence to construct the contribution story.

Step 5 – Seek additional evidence

Address gaps in the initial contribution story by gathering more evidence.
Use surveys, case studies and other research methods to enhance the robustness of the analysis.

Step 6 – Revise and strengthen the contribution story

Refine the contribution story based on new evidence.
Reassess strengths and weaknesses, and adjust the theory of change if necessary.

A contribution analysis works best as an iterative process. Thus, at this point the analysis may return to Step 4 and reassess the strengths and weaknesses of the contribution story.

Main takeaway points

A contribution analysis provides a clear process for exploring cause-and-effect relationships.
It is a method suitable for various contexts, especially where experimental designs are not feasible.
It allows for continuous refinement and strengthening of an analysis.
Contribution analyses incorporate multiple sources of evidence, including qualitative and quantitative data. Therefore, it builds a robust narrative that can withstand scrutiny.
The method requires significant time and effort to gather and analyse evidence.

Learning from experience

Horton, D., Mackay, R., Andersen, A. and Dupleich, L. (2000)
Evaluating capacity development in planning, monitoring, and evaluation: a case from agricultural research. Research Report no. 17
Kotvojs, F., (2007)
Contribution analysis: a new approach to evaluation in international development. Evaluation Journal of Australasia, Volum 7, Issue 1
Mayne, J., (2001)
Addressing attribution through contribution analysis: using performance measures sensibly. Canadian Journal of Program Evaluation 16: 1-24
Morton, S., (2015)
Progressing research impact assessment: A ‘contributions’ approach. Research Evaluation, Volume 24, Issue 4, October 2015, p. 405–419
Riley, B., Kernoghan, A., Stockton, L., Montague, S., Yessis J., Willis C., (2017)
Using contribution analysis to evaluate the impacts of research on policy: Getting to ‘good enough’. Research Evaluation, Volume 27, Issue 1, January 2018, p. 16–27