Learning portal - Naïve group comparisons

A pragmatic alternative

Naïve group comparisons, supported by qualitative methods, are suggested as an alternative for assessing impacts. They can act as a ‘quick fix’ when there is neither time to set up a proper survey nor existing monitoring data to use with a counterfactual or estimate a programme’s net effects through a sound, statistical methodology.

In this evaluation technique, the necessary data on the average values of outcome indicators for units not participating in a programme are usually obtained from various national surveys or aggregated national data. The approach relies on the assumption that in the absence of a programme, the value of the outcome indicator of programme participants would be the same as the average of a joint group of programme participants and non-participants. This, however, would only be justifiable if the performance of a group of programme participants (measured by any arbitrary impact indicator, e.g. income, profit or employment) was identical to the performance of a joint group of programme participants and non-participants (population average).

Quick and easy, but be cautious

Naïve estimates: with this approach, the comparison groups are usually selected arbitrarily, leading to quantitative results that are statistically biased. Evaluations sometimes use this less robust approach, which does not allow for as rigorous knowledge about a specific programme’s direct and indirect effects if they do not have sufficient data and control groups. A naïve estimate implies applying methods based on insufficient evidence or ad hoc surveys of a group of beneficiaries, opinions of administrative officials, etc. These techniques are generally unsuitable to appropriately address the issues considered crucial in any quantitative evaluation framework.

Naïve comparison techniques can be used when only a few units participate in a programme and, hence, effects are not expected. This technique can also be applied when there is neither data about observable variables or instruments explaining programme participation and outcomes nor appropriate structural models and other modelling approaches.

Restrictions concerning the interpretation of calculations based on this simplified technique should be considered, especially regarding the magnitude of a possible selection bias.

The technique can be applied to assess the effect of CAP support on the evolution of the values of the impact indicators listed in the following table.

Step 1 – Construct the average of the change in the outcome indicators for units participating in a programme.

Step 2 – Set up the ‘counterfactual’, which is the corresponding average of the NUTS 2 area or other broader area in which participating units are located.

Step 3 – Estimate a ‘net’ effect by comparing the average of the participants (Step 1) to the counterfactual (Step 2).

Step 4 – Apply naïve difference in differences (DiD) method if, from monitoring data, the evaluator can calculate the average values of the outcome indicators before and after a programme's implementation.