Learning portal - Propensity score matching-difference in differences (PSM-DiD) method

• Abundant data on programme participants and non-participants prior to and after a programme (panel data).
• High quantitative skills of the evaluator.
• Good understanding of conditions determining the probability of participation in a programme.

The technique can be applied to assess the effect of CAP support on the evolution of the values of the impact indicators listed in the following table.

Specifically for the Farmland Bird Index, an assessment approach conducted at the field/plot scale level (micro level) can be realised by using the Common Birds Monitoring Programme if sufficient data is available. This can be achieved by applying the PSM-DiD technique to match beneficiaries and non-beneficiaries and then comparing the average CAP support effect on biodiversity in each group before and after the implementation of a programme. At the macro level, PSM is also recommended to net out the CAP support effect on biodiversity at the level of quadrants used to observe the farmland birds’ populations under the Common Birds Monitoring Programme. The quadrants can be used as functional units for the Farmland Bird Index and then calculated by bio-geographical areas (different agricultural habitats) or at the regional level based on geo-referenced data.

Regarding indicators measuring employment, GDP per capita and poverty rate in rural areas, the beneficiaries’ and control groups can be constructed based on supported and non-supported geographical regions, ideally LAU 2 as defined by the Eurostat urban-rural typology. In this case, access to a comprehensive source of statistical data and information about characteristics of geographical regions before and after the implementation of the programme is required.

• Step 1 – Find a sample of beneficiaries (e.g. farms, farmers, non-farming enterprises, communities, areas, regions etc.) in an available database (e.g. FADN) and use the monitoring and evaluation electronic system (e.g. the electronic information system of Article 70, Regulation (EU) 1305/2013, or the one of Article 130, Regulatio (EU) 2021/2115) as a reference point.
• Step 2 – In the same database (e.g. FADN), select all relevant units that did not receive support or received an arbitrary low level of support (non-beneficiaries) in the same period. In the latter case of arbitrary low level of support, it might be better to define the low level by not using absolute levels of support but rather the levels of support normalised by another variable, like the utilised agricultural area of the farmer or the area or population of a geographical region.
• Step 3 – In a group of non-beneficiaries, identify those units which could not fulfil the programme eligibility conditions (due to high income, size, location, etc.) and remove them from the analysis.
• Step 4 – Collect data for all units in both groups (beneficiaries and non-beneficiaries) on their significant characteristics (variables) at the start of an implementation period. Note that the variables included in the analysis should affect both the selection of a unit and the indicators computed at the micro-level (common and additional impact indicators). Some of the proposed variables (used as important control variables) can be:
  • the level of support received by a given unit during previous programming periods; and/or
  • the level of support received by a given unit from other public sources (e.g. EU structural funds, Pillar I) in the analysed period.
• Step 5 – Apply appropriate techniques to identify a suitable control group from the sample of non-beneficiaries (see Steps 2-3), the members of which have the same propensity to participate in a programme (some of the non-beneficiaries and/or beneficiaries will be dropped from the analysis due to a lack of adequate control units).
• Step 6 – Check statistically the similarity of both groups prior to receiving support from a programme (e.g. by performing statistical tests on covariates included in the analysis). The average value of a unit in the beneficiary group should not significantly differ from the respective unit in the control group. Once the beneficiaries’ group and the control group have been constructed, the net effect of the support can be estimated through the subsequent steps.
• Step 7 – Compute the average value of outcome indicator(s) for the group of beneficiaries and the control group before the support and when the evaluation is carried out.
• Step 8 – Calculate the difference in the average value of the outcome indicator(s) for the beneficiaries’ group before the support and at the time of the evaluation (first difference).
• Step 9 – Calculate the difference in the average value of the outcome indicator(s) for the non-beneficiaries group before the support and at the time of the evaluation (second difference).
• Step 10 – Calculate the net effect of CAP support by subtracting the second difference (Step 9) from the first difference (Step 8).
• Step 11 – Perform a sensitivity analysis (e.g. Rosenbaum bounding approach) to assess the possible effects of unobservables on obtained results.
• Step 12 – Aggregate the findings and calculate the CAP support effects on the analysed impact indicators at the macro and programming area levels. In this step, the evaluator should calculate net direct effects of the CAP support on the impact indicators at the programme area level by applying extrapolation techniques (i.e. by multiplying average micro-results computed at a micro level by the number of beneficiaries/non-beneficiaries).

• PSM-DiD is very effective in controlling both observable and unobservable selection bias.
• The method involves matching participants with similar non-participants, and then measuring changes in impact indicators over time.
• It is most effective in situations with comprehensive data and requires an evaluator with a strong quantitative background.
• PSM-DiD offers detailed insights into the effectiveness and efficiency of interventions.
• The success of PSM-DiD depends on the depth of the available data and the methodology used for matching and comparison.

• Caliendo, M., & Kopeinig, S. (2008)
  Some Practical Guidance for the Implementation of Propensity Score Matching. Journal of Economic Surveys, 22(1), 31-72

• European Evaluation Network for Rural Development for the CAP (2014)
  Capturing the success of your RDP: Guidelines for the ex post evaluation of 2007-2013 RDPs