Learning Portal

Learning portal - Most significant change method

The most significant change (MSC) method is a transformative approach to evaluation. It captures personal stories of change from various levels of an intervention's governance. It's unique for its hierarchical story selection and active involvement of stakeholders.

Hand flipping wooden cube block

Basics

In a nutshell

Storytelling at the core

The most significant change (MSC) method involves the collection of personal accounts of change (stories or outcomes) and determining which of them are the most significant through a systematic, hierarchical selection process involving stakeholders from all levels of governance related to an intervention. By involving stakeholders at different levels of governance, indications about what is valued by each level also emerge. The selected stories, along with the criteria for their selection, are then communicated to all the other levels to create a common understanding of the most positive or negative outcome of an intervention.

To be combined with other methods

The MSC method is not just about providing evidence of the usual experience. It's about revealing extremes and facilitating the improvement of the intervention being evaluated. It can help organizations focus their work towards explicitly valued directions and away from less valued ones. It also provides a wealth of mini-case study material to support and illustrate arguments on how change comes about (e.g. how farmers change their practices or how collaboration comes about) and when it occurs (e.g. after an AKIS intervention, as a result of it or as a result of a contextual change).

Pros and cons

Advantages

Disadvantages
  • Enables the improvement of the interventions and the generation of knowledge on what works and for whom.
  • Identifies unexpected outcomes.
  • Does not require special professional skills.
  • Builds stakeholders’ capacity to analyse data and impacts.
  • Supports bottom-up initiatives that do not have predefined outcomes against which to evaluate.
  • Time-consuming with multiple cycles needed.
  • Less suited for capturing expected changes.
  • Not ideal for retrospective evaluations.

When to use?

MSC can be used in cases where conventional monitoring and evaluation tools may not provide sufficient data to make sense of program impacts and foster learning. These may include interventions that are:

  • complex and produce diverse and emergent outcomes;
  • large with numerous organisational layers;
  • focused on social change;
  • designed with repeated contact between the implementing body(ies) and beneficiaries;
  • struggling with conventional monitoring systems;
  • highly customised services to beneficiaries (such as farm advisory services).

Preconditions

  • Engagement from different groups involved in the process and maintaining their interest.
  • Good facilitation skills and ability to identify priorities.
  • Time availability (for the analysis of stories and sharing with contributors and stakeholders, repeated through several cycles).
  • Evaluators with experience in using MSC.

In the context of the CAP, this method can be used, in conjunction with other qualitative and quantitative methods, for the assessment of the following types of interventions:

  • Cooperation, including LEADER.
  • Knowledge exchange and dissemination of information.

It can also be used to triangulate the results of quantitative analysis of most other types of interventions and to identify unintended outcomes.

Finally, it can be used in the preparation of CAP evaluations. For example, the criteria used by the different levels to select the MSC stories can inform the formulation of factors of success.

Step-by-step

  • Step 1 – Introduce MSC to a range of stakeholders and foster interest and commitment to participate.
  • Step 2 – Identify the domains of change to be monitored. This involves selected stakeholders identifying broad domains, such as changes in people’s quality of life that are not precisely defined, like performance indicators, which are deliberately left loose to be later defined by actual beneficiaries.
  • Step 3 – Decide how frequently to monitor changes in these domains.
  • Step 4 – Collect significant change stories from those most directly involved, such as participants and implementing staff. The stories are collected by asking a simple question such as: ‘During the last month, in your opinion, what was the most significant change that took place for participants in the program?’. It is initially up to respondents to allocate their stories to a domain category. In addition to this, respondents are encouraged to report why they consider a particular change to be the most significant one.
  • Step 5 – Select the most significant change stories by analysing and filtering them up through the levels of governance typically found within an intervention. Each level reviews a series of stories sent to them by the level below and selects the single most significant account of change within each of the domains. Each group then sends the selected stories up to the next level of the program hierarchy, and the number of stories is whittled down through a systematic and transparent process. The criteria used to select stories are recorded every time they are selected.
  • Step 6 – Feed the selected stories and the criteria used to select them back to all interested stakeholders so that feedback from previous rounds informs each subsequent round of story collection and selection. The delivery mechanism effectively records and adjusts the direction of its attention – and the criteria it uses to evaluate the events it sees there. After this process has been used for some time, such as a year, the Managing Authority may produce a document with all stories that best represent the outcomes that contribute most to the objectives of the intervention. The stories are accompanied by the reasons the stories were selected. This information is fed back to all stakeholders involved.
  • Step 7 – Verify the selected stories by visiting the sites where the described events took place. The purpose of this is two-fold: to check that stories have been reported accurately and honestly, and to provide an opportunity to gather more detailed information about events seen as especially significant. If conducted some time after the event, a visit also offers a chance to see what has happened since the event was first documented.
  • Step 8 – Quantify the results. When an account of change is first described, it is possible to inclue quantitative and qualitative information. It is also possible to quantify the extent to which the most significant changes identified in one location have taken place in other locations within a specific period.
  • Step 9 – Perform a secondary analysis by examining who participated, how they affected the contents and how often different types of changes are reported.
  • Step 10 – Revise the design of the MSC process to take into account what has been learned as a direct result of using it and from analysing its use.

Main takeaway points

  • Stakeholders are centrally involved in the MSC method, bringing diverse perspectives to the evaluation.
  • MSC focuses on significant, often unexpected, outcomes rather than typical experiences.
  • Different from conventional methods, MSC captures extremes to inform and guide interventions.
  • Ideal for complex scenarios, social change initiatives and situations requiring tailored monitoring.
  • The MSC process is a systematic journey from story collection to analysis, feedback, and improvement.

Learning from practice

Further reading