Learning Portal

Learning Portal - Difference in differences method

The difference in differences (DiD) method is a quasi-experimental evaluation methodology focusing on changes in programme effects. It compares the effects on participants before and after a programme with those on non-participants, eliminating biases to isolate the net effect of the programme.

a wheat field with numbers and circles

Basics

In a nutshell

The difference in differences (DiD) method is a quasi-experimental evaluation methodology.

The technique compares the changes in the programme effects of programme participants, before and after the implementation of the programme, with the corresponding changes for arbitrary selected non-participants.

The assumption behind

The main assumption behind this technique is that the changes in the programme effects on both participants and non-participants would be very similar in the absence of the programme. Therefore, if the changes observed by non-participants are subtracted from the ones observed by participants, the part of the change that would have happened even in the absence of the programme is eliminated, and what remains is the net effect of the programme.

Estimating programme effects is carried out in two steps:

  1. calculating the average difference in outcome indicators separately for programme participants and non-participants; and
  2. establishing the difference between the average changes in outcome indicators for these two groups.

The difference in the before and after outcomes for the programme’s supported beneficiaries (the first difference) controls for factors that are constant over time in that group. After subtracting the second difference (in a control group), the technique may eliminate the main source of bias (i.e. the change that would have happened anyway), which was present in the basic before and after comparisons. The DiD approach thus combines two ‘naïve’ techniques i.e. before and after comparisons of programme beneficiaries, and comparisons between beneficiaries with non-beneficiaries to produce a better estimate of a counterfactual.

With this method, any common trend in the outcomes of programme participants and non-participants (selection bias) gets filtered out, but the key assumption justifying this method is that selection bias remains constant over time (the so-called fixed effect). In consequence, the DiD method cannot help to eliminate differences between programme beneficiaries and non-beneficiaries that change over time.

Under the DiD method, a group of programme beneficiaries does not need to have the same pre-support conditions as a group of non-beneficiaries. However, for DiD to be valid, a group of programme non-beneficiaries must accurately represent change in outcomes that programme beneficiaries would have experienced in its absence.

Pros and cons

Advantages

Disadvantages

  • Allows to control unobserved heterogeneity (which may impact outcomes), but only under the assumption that this does not vary in time.
  • Flexible approach allowing for illustrative interpretation.
  • Easily combined with matching estimators to allow for better accuracy of results.
  • Relies on the assumption of similar outcome trends for participants and non-participants in the absence of the programme. If, however, outcome trends are different for the participants and non-participants, then the estimated treatment effect obtained by the DiD method would be invalid or biased.
  • Generally less robust than other quasi-experimental methods (e.g. PSM or GPSM). The common trend assumption might not be verified or testable.

When to use?

The DiD method can be used when the available data allows estimating a programme effects on programme participants and non-participants, before and after the implementation of a programme, but it does not allow for comparing the two groups based on the similarity of their characteristics.

The approach can be used under the condition that, without a programme, outcomes would increase or decrease at the same rate for both participants and non-participants of a programme. A good validity check for this assumption is to compare changes in outcomes in both groups in a longer period, before a programme was implemented. This, however, does not imply that only simple aggregated average values of beneficiaries and non-beneficiaries should be compared. Instead, a careful design of pairwise comparisons and multiple comparison groups differentiated by known factors and observables accompanied by the DiD method should be applied to reduce bias.

Overall, the available evidence shows that standard DiD estimators’ ability or performance may not be a sufficient choice in many evaluation contexts.

Preconditions

  • The method requires either longitudinal or repeated cross-sectional data (time series of cross-sectional data) on outcome indicators collected for programme beneficiaries and non-beneficiaries.
  • Strong evidence that in the absence of a programme, the performance of the programme beneficiaries and non-beneficiaries will be the same (or similar).
  • The support must have occurred between two periods observed by the researcher.

The technique can be applied to assess the effect of CAP support on the evolution of the programme effects listed in the following table.

RDP impact indicator CAP Strategic Plan impact indicator
I.01 - Agricultural entrepreneurial income I.2 - Evolution of agricultural income compared to the general economy
I.02 - Agricultural factor income I.3 - Evolution of agricultural income
I.03 - Total factor productivity in agriculture I.6 - Total factor productivity in agriculture
I.07 - Emissions from agriculture

I.10 - Greenhouse gas emissions from agriculture

I.14 - Ammonia emissions from agriculture

I.08 - Farmland bird index I.19 - Farmland bird index
I.09 - High nature value (HNV) farming  
I.10 - Water abstraction in agriculture I.17 - Water Exploitation Index Plus (WEI+)
I.11 - Water quality

I.15 - Gross nutrient balance on agricultural land

I.16 - Nitrates in ground water

I.13 - Soil erosion by water I.13 - Percentage of agricultural land in moderate and severe soil erosion
I.14 - Rural employment rate I.24 - Evolution of the employment rate in rural areas, including a gender breakdown
I.15 - Degree of rural poverty I.27 - Evolution of poverty index in rural areas
I.16 - Rural GDP per capita I.25 - Evolution of gross domestic product (GDP) per capita in rural areas

Step-by-step

  • Step 1 – Construct comparison groups of beneficiaries and non-beneficiaries, differentiated by known factors and observables, for which the value of the corresponding impact indicator can be calculated.
  • Step 2 – Calculate the average value of the impact indicator for beneficiaries at the start of the implementation period.
  • Step 3 – Calculate the average value of the impact indicator for beneficiaries at the time of the evaluation.
  • Step 4 – Calculate the first difference for beneficiaries before (Step 1) and after the implementation of the intervention(s) (Step 2).
  • Step 5 – Calculate the average value of the impact indicator for non-beneficiaries, at the start of the implementation period.
  • Step 6 – Calculate the average value of the impact indicator for non-beneficiaries at the time of the evaluation.
  • Step 7 – Calculate the second difference for non-beneficiaries before (Step 5) and after the implementation of the intervention(s) (Step 6), in each group constructed in Step 1.
  • Step 8 – Calculate the DiD by subtracting the second difference (Step 7) from the first difference (Step 4).

Main takeaway points

  • The DiD method carefully eliminates biases by comparing programme effects before and after implementation among participants and non-participants.
  • Its foundational assumption is that outcome trends for both groups would be similar without the programme.
  • DiD is most effective when data is available for both groups over time and the assumption of the trend similarity holds.
  • This approach compares changes for programme participants and non-participants to isolate the programme's effect.
  • Key steps in the DiD method include forming comparison groups, calculating average changes and deducing differences between the groups.

Learning from practice

Further reading

Publication - Guidelines and tools |

Assessing RDP Achievements and Impacts in 2019