Learning Portal - Difference in differences method

The difference in differences (DiD) method is a quasi-experimental evaluation methodology.

The technique compares the changes in the programme effects of programme participants, before and after the implementation of the programme, with the corresponding changes for arbitrary selected non-participants.

The assumption behind

The main assumption behind this technique is that the changes in the programme effects on both participants and non-participants would be very similar in the absence of the programme. Therefore, if the changes observed by non-participants are subtracted from the ones observed by participants, the part of the change that would have happened even in the absence of the programme is eliminated, and what remains is the net effect of the programme.

Estimating programme effects is carried out in two steps:

1. calculating the average difference in outcome indicators separately for programme participants and non-participants; and
2. establishing the difference between the average changes in outcome indicators for these two groups.

The difference in the before and after outcomes for the programme’s supported beneficiaries (the first difference) controls for factors that are constant over time in that group. After subtracting the second difference (in a control group), the technique may eliminate the main source of bias (i.e. the change that would have happened anyway), which was present in the basic before and after comparisons. The DiD approach thus combines two ‘naïve’ techniques i.e. before and after comparisons of programme beneficiaries, and comparisons between beneficiaries with non-beneficiaries to produce a better estimate of a counterfactual.

With this method, any common trend in the outcomes of programme participants and non-participants (selection bias) gets filtered out, but the key assumption justifying this method is that selection bias remains constant over time (the so-called fixed effect). In consequence, the DiD method cannot help to eliminate differences between programme beneficiaries and non-beneficiaries that change over time.

Under the DiD method, a group of programme beneficiaries does not need to have the same pre-support conditions as a group of non-beneficiaries. However, for DiD to be valid, a group of programme non-beneficiaries must accurately represent change in outcomes that programme beneficiaries would have experienced in its absence.

The DiD method can be used when the available data allows estimating a programme effects on programme participants and non-participants, before and after the implementation of a programme, but it does not allow for comparing the two groups based on the similarity of their characteristics.

The approach can be used under the condition that, without a programme, outcomes would increase or decrease at the same rate for both participants and non-participants of a programme. A good validity check for this assumption is to compare changes in outcomes in both groups in a longer period, before a programme was implemented. This, however, does not imply that only simple aggregated average values of beneficiaries and non-beneficiaries should be compared. Instead, a careful design of pairwise comparisons and multiple comparison groups differentiated by known factors and observables accompanied by the DiD method should be applied to reduce bias.

Overall, the available evidence shows that standard DiD estimators’ ability or performance may not be a sufficient choice in many evaluation contexts.

Preconditions