Evaluation FAQ

There is no requirement on the number of evaluations, it is up to the EU Member State to decide how many evaluations it will undertake.

According to Article 2(a) of Implementing Regulation (EU) 2022/1475, EU Member States are required to evaluate each Specific Objective and Cross-Cutting Objective (CCO) of modernisation during the implementation of the CAP Strategic Plan in line with its intervention logic. Where applicable, the AKIS strategic approach should be assessed based on the EU Member State's evaluation needs, taking into account the intervention logic and implementation of the CAP Strategic Plan, as stated in Article 2(d) of the Implementing Regulation. This implies that the evaluation (scope, focus and timing) of the AKIS strategic approach will be defined by the EU Member State's evaluation needs, intervention logic of the AKIS strategic approach, and its implementation.

If the implementation of AKIS related interventions in an EU Member State is shared responsibility between the national and regional levels, the interventions implemented at regional level can also be analysed at the relevant level, but the findings/conclusions can be aggregated at the national level to provide evidence on the achievement of the CAP Strategic Plan.

The CAP Strategic Plan regulation only requires an evaluation at the CAP Strategic Plan level. If the EU Member State considers it relevant, it can also decide to conduct several AKIS evaluations at the regional level or assess the coordination between the national and regional level. EU Member States also need to take into account recital 120 of Regulation (EU) 2021/2115 which encourages them to ensure the involvement of the regions in the design of the evaluation plan and in the monitoring and evaluation of the regional interventions of the CAP Strategic Plan.

To support EU Member States in the evaluation of AKIS, the guidelines ‘Evaluating the AKIS strategic approach in CAP Strategic Plans’ have been developed by the Evaluation Helpdesk and will be published on the EU CAP Network website in May 2023.

No, it is not necessary to conduct separate evaluations. In line with Article 114(a) of the CAP Strategic Plan Regulation (EU) 2021/2115, AKIS is expected to contribute to the achievement of the CCO set out in Article 6(2), so it can be evaluated in the context of the CCO evaluation. Given that AKIS strategic approach will be evaluated as part of the CCO assessment, the requirement of the Article 2(d) of the Implementing Regulation (EU) 2022/1475 will be met respectively.

However, based on EU Member State evaluation needs, more focused evaluation of AKIS strategic approach can be performed, for example on the functionality of AKIS or addressing more specific elements, such as knowledge flows and strengthening links between research and practice.

To support EU Member States in the evaluation of AKIS, the guidelines ‘Evaluating the AKIS strategic approach in CAP Strategic Plans’ have been developed and will be published on the EU CAP Network website in May 2023.

In paragraph 1 of Article 1 of the Implementing Regulation (EU) 2022/1475 it is written that “[w]hen evaluating their CAP Strategic Plans, Member States shall define evaluation questions and factors of success to assess the evaluation criteria of effectiveness, efficiency, relevance, coherence and Union added value referred to in Article 140(1) of Regulation (EU) 2021/2115”.

Then, a distinction is made between evaluations conducted during the implementation of the CAP Strategic Plans and those conducted ex post. For the former, according to point (b) of Article 2 of the above regulation, “Member States shall assess their CAP Strategic Plans using the relevant evaluation criteria” and according to point (e) “Member States shall carry out evaluations in time to be able to prepare the subsequent CAP Strategic Plan’s period”. The ex post evaluations “shall include the assessment of the CAP Strategic Plans and their implementation based on each of the evaluation criteria of effectiveness, efficiency, relevance, coherence and Union added value” (paragraph 2 of Article 3).

Information on how and when to incorporate the evaluation criteria in the evaluation plan and framework can be found in the Better Regulation Toolbox and in the evaluation plan guidelines published in March 2023 by the Evaluation Helpdesk.

Regarding the relative weight of each evaluation criterion, in Tool #47 of the Better Regulation Toolbox it is mentioned that “[t]he degree of analysis conducted for each criterion should be proportionate: it depends on the intervention being evaluated, the timing of the evaluation and the adequacy and reliability of the data”. Moreover, according to OECD, the decision can be based on two principles:

Article 140(2) of Regulation (EU) No 2021/2115 states that “Member States shall entrust evaluations to functionally independent experts”. Evaluations shall be carried out by internal or external experts that are functionally independent of the authorities responsible for programme implementation.

As reference, during the previous programming period (2014-2020), the ‘Guidance Document on Monitoring and Evaluation for the programming period 2014-2020‘ issued by DG Regional Policies in March 2014 clarifies that the level of independence should be such that there is no doubt that the work is carried out with objectivity, and the evaluation judgments are unbiased and not subordinated to an agreement of the services responsible for conducting the programme.

Therefore, the European Commission services consider it as best practice to assign the evaluation to external experts or to a different organisation from that responsible for managing/implementing the programme and any of the intermediate bodies reporting to it.

Finally, where evaluation and programme implementation are assigned to the same department or unit of an organisation, this could give rise to doubts about “functional independence” of the evaluation. Hence, clear arrangements for ensuring independence, though, for instance, the following measures should be made:

The aim is to assess the effects and interplay of whole set of green elements of the CAP. At least Conditionality and requirements to receive CAP support and green measures (eco-schemes, AEM, sectoral interventions, etc.) which are aimed to, or focus on, environment (no secondary effects, but clearly and actually aimed to address environmental-related needs/objectives/aspects). See Meeting highlights - Workshop on Implementing the CAP Green Architecture (europa.eu).

The assessment of the Green Architecture is done during the implementation period as well as during the ex post evaluation where it is part of the overall assessment of General Objectives. The assessment can focus both on the provisions/building stones of Green Architecture in the CAP Strategic Plan (earlier) as well as on the respective effects/impacts (later in the programming period). The assessment of the Green Architecture is carried out during the implementation period as well as during the ex post evaluation where it is part of the overall assessment of General Objectives.

The indicators that will be used in each evaluation should be defined when developing the evaluation framework. Their selection would typically depend on the evaluation questions, which should reflect the underlying intervention logic, and how these would be answered through the corresponding factors of success. Each factor of success should be linked with appropriate indicators that reflect the causal chain from outputs and results to impacts.

The decision about which result indicators should be linked to the factors of success and used in the evaluation, should follow the intervention logic established in the CAP Strategic Plan. Those result indicators to which the underlying interventions contribute directly and significantly, should be typically considered in the causal chain. This direct and significant contribution is not determined only by the relative budget size. For example, even if the production of renewable energy, although on a small-scale, or the promotion of digital farming technologies represent a small part of a single large on-farm modernisation investment, this intervention should nevertheless be linked to R.3 (digitalising agriculture) and the installed capacity be reported under R.15 (renewable energy).

Values of the CAP context (and impact) indicators are published on the European Commission’s agri-food data portal in the section ‘CAP indicators’.

There you can find the latest available values for CAP indicators following the context indicators CMEF code (context indicators that incorporate impact indicators are marked with an asterisk(*)). Context and impact indicator fiches (Version 8.0 published on Circa 14/02/2022) provide a table that allows the identification of the respective CAP indicators for the 2023-2027 programming period with the respective PMEF indicator codes.

The Commission will continue to publish values of CAP indicators as soon as they will become available from Eurostat or other relevant data sources.

The list of indicators is also foreseen to be updated by aligning them with the PMEF, adding new indicators that are listed in the Annex I of the CAP Strategic Plan Regulation (EU) 2021/211 and adding updated fiches of these indicators. In the context and impact indicator fiches, EU Member States can find information on most of the impact indicators about the frequency at which their values are collected/calculated and timeliness (how old are the data when they become available).

Additionally, I.21 (LUCAS data 2022) is currently being analysed by the JRC and should be available in the course of 2023, while

I.22 (Data (2020)) be available for a baseline in 2023.

Although values of context/impact indicators are provided by the Commission, the contribution of the relevant CAP Strategic Plans to these common impact indicators is not always evident and these values are not enough to carry out evaluations at the EU Member State level. Therefore, EU Member States shall quantify the contribution of the CAP Strategic Plans of at least the common impact indicators set out in Annex III of the Implementing Regulation (EU) 2022/1475.

Stakeholders mapping allows a Managing Authority to collect all the information needed to develop a coherent strategy for stakeholder engagement in the evaluation process. This will help the Managing Authority identify all relevant stakeholders and understand how and when they should be engaged with, as well as understand their specific characteristics and needs, classification according to their interest in, and power to, affect the implementation and evaluation of the CAP Strategic Plan interventions.

For example, the Managing Authority should collaborate closely, throughout the evaluation process, with stakeholders with high interest and power to define the scope of the corresponding evaluations, including evaluation questions, and their timing, such as the Paying Agency, intermediate bodies or strong farmer unions. High interest and low power stakeholders, like LAGs, farm advisors or environmental NGOs, could be involved to further refine the scope of the evaluations during the initial stages and/or at the end of the process to check whether evaluation findings are grounded in reality.

High power and low interest stakeholders, like influential researchers and academics, can be consulted at the initial stages regarding the validity of the evaluation design and/or the robustness of the methods that could be devised to answer evaluation questions. Additionally, depending on their level of engagement, different stakeholder groups may have different needs for capacity building.

The concept of stakeholder mapping is addressed extensively in the evaluation plan  Guidelines: Design of evaluation plans, published in March 2023 by the Evaluation Helpdesk. In this publication, beyond a specific chapter for stakeholder mapping, there are several tools including how to identify and classify stakeholders and how to assess their needs for capacity building.

According to the Implementing Regulation (EU) 2022/1475, “where relevant, based on Member States’ evaluation needs and taking into account the CAP Strategic Plan’s intervention logic and implementation, Member States shall assess also specific interventions or topics of the CAP Strategic Plans, such as the environmental and climate architecture […]” (point d of Article 2). At the same time, “Member States shall plan the evaluations of the specific objectives […] which are addressed in their CAP Strategic Plans, in accordance with the CAP Strategic Plan’s intervention logic, by objective or by comprehensive evaluations covering several objectives […]” (point a of Article 2).

Based on the above, the assessment of interventions covering several objectives could be either part of a topic-specific evaluation or part of an evaluation covering several Specific Objectives (SO). Considering eco-schemes’ interventions, the decision could be based on the intervention logic of each relevant SO. If, for example, SO1 includes interventions that could be considered as not being part of the green architecture (such as risk management tools) then assessing eco-schemes as part of an evaluation grouping SO1, 4 and 5 might be less appropriate than including this in a topic-specific evaluation about green architecture. In any case, the analysis conducted for the assessment of eco-schemes, or other interventions that contribute to multiple objectives, as part of a topic-specific evaluation, can be reused in the framework of an evaluation of a group of objectives and vice versa.

What might be important for interventions contributing to multiple objectives is to set up the monitoring system in a way that allows the tracing of the distinct causal paths. For example, an evaluation of an eco-scheme intervention, comprising of several commitments with differentiated contribution to the different specific objectives, establish links between each commitment and the corresponding objective(s) in the monitoring system, allowing the tracing of the contribution of each single operation to the various objectives. This could be even more relevant for investment interventions which include, for example, an environmental component along with a farm modernisation one. In this case, explicit links between categories of expenditure and corresponding objective(s), could allow not only the tracing of the contribution of each single operation to the various objectives, but also the tracing of when this contribution has been realised – based on when the payment to the corresponding categories of expenditure was executed. When establishing these links, the Managing Authority should always have in mind the relationships between types of interventions and result indicators, as well as result indicators and SOs described in the cover note on output and result indicators.