Learning Portal

Learning portal - Surveys

Surveys are particularly effective in drawing conclusions about a larger population from a smaller, representative sample. Their main advantage lies in bridging data gaps and providing a comprehensive view of a subject. This approach is engaging and insightful, allowing for a deeper understanding of complex topics.

Note pinned on a board

Basics

In a nutshell

Surveys are mixed methods (qualitative and quantitative) that are often part of a deductive analytical approach. They deduce information from a (representative) sample to depict the reality of the total. In a deductive approach, hypotheses are mostly built from literature or a theoretical framework before surveys are designed. The data collected are then examined to test the theory built a priori.

Sampling for the bigger picture

In a survey, the information is gathered from a sample of entities for the purpose of constructing quantitative descriptors of the attributes of the larger population of which the entities are members.

Surveys are to be applied in various circumstances of the assessment of effects, independently on the type of information to be collected (e.g. sampling of control groups), the type of indicator it is serving (social science adopts different survey methods than natural science) and programme specifics (data availability, type of intervention etc.).

Feeding into other methods

Surveys can be very valuable in collecting quantitative data (e.g. basic characteristics and performance of supported and non-supported units), which can be used later to estimate programme results, e.g. via matching techniques. While using this data collection technique, particular attention has to be paid to the accuracy of information concerning past events e.g. characteristics and performance of programme-supported and non-supported units in a time period before a given programme is implemented.

A representative survey of programme components can be used with questionnaire-based interviews of programme beneficiaries and non-beneficiaries, in-depth interviews, and focus groups to gather evidence of programme effects. Under this approach, focus group discussions may help to identify possible sources of bias in a quantitative survey and a way to address them.

Not a one-size-fits-all method

A vast amount of literature is available to identify an appropriate survey methodology, but no ‘one-size-fits-all’ approach exists. Thus, applying surveys related to different thematic fields of indicators has to be deliberately selected and adjusted accordingly.

Pros and cons

Advantages

Disadvantages
  • Collects primary data, offering fresh insights.
  • Bridges data and information gaps effectively.
  • Capable of gathering both qualitative and quantitative data.
  • Represents an ad hoc tool to collect data and information.
  • Risk of bias from selected samples of beneficiaries/non-beneficiaries.
  • Risk of outliers.
  • Requires significant resources.

When to use?

In the context of a CAP Strategic Plan, an assessment survey can help collect primary data, in particular for:

  • Specific characteristics of CAP beneficiaries/non-beneficiaries. The survey will help establish a typology of CAP beneficiaries/non-beneficiaries. This might be particularly needed to understand the drivers behind farmers’ decisions to apply to CAP interventions (notably agro-environmental measures) and/or implement sustainable farm practices. Hence, such a survey could bring useful information for a critical review of the theoretical intervention logic and explain low uptake rates of specific CAP interventions. The distributed questionnaire must offer a wide range of choices identified from preliminary literature research and include the option to write down other drivers not yet identified.
  • Strategy/options implemented by farmers/CAP beneficiaries. A survey can help determine to what extent farmers face specific issues and how they address them with or without the CAP. This is important to understand farmers’ needs and appraise the complementarity of CAP interventions with other available tools (e.g. private market tools). For instance, a survey can collect information on the impact of market fluctuations on farmers’ revenues and their strategy to overcome significant price drops. It can ask respondents to indicate the severity of the impact by granting a score between 1 (no impact) and 5 (most severe impact) and examine the different types of strategies and risk management tools implemented by farmers, as well as the potential use of CAP interventions (e.g. support for risk management). The results can then be used to classify which cases and how the CAP interventions are used according to farmers' exposure to market volatility and the strategy implemented.
  • Costs associated with the implementation of specific CAP interventions. An assessment of CAP Strategic Plan efficiency will require appraisal of the administrative burden and other costs (e.g. transaction costs) associated with implementing CAP interventions. Information on administrative and/or monitoring costs or other types of costs borne by farmers are generally unavailable and need to be collected in the CAP Strategic Plan assessment frame. Questionnaires containing close-ended questions allow the collection of quantitative or easily quantifiable information. Judgments and opinions can be collected through multiple-choice questions with pre-coded answers and a Likert scale.
  • Surveys can also be used to collect information on the effectiveness of a CAP Strategic Plan. However, the effects should be examined through a mix of methods that allow for the assessment of the net contribution of the CAP Strategic Plan. The survey methodology must be carefully established to provide valid results on the net effects of measures implemented. Moreover, when asked about the effects of CAP support, CAP beneficiaries can provide biased or erroneous responses that should be considered with caution. Nevertheless, when no data is available on the impact of CAP interventions, structured written surveys can be used to gather information on the ‘changes’ the farmers carried out to participate in CAP support schemes, to provide useful insights and a proxy indication of the effects achieved by the CAP interventions. Changes can be measured through specific quantitative indicators established on an ad-hoc basis, and data can be collected through questionnaires sent to the targeted stakeholders.

Step-by-step

Conducting a survey entails three stages: preparation and question design, administration of the survey, and analysis of results

  • Step 1 – Preparation and question design: The questionnaire's content must reflect the survey's scope and purpose. In the context of a CAP Strategic Plan assessment, the questionnaire should be consistent with the evaluation framework and enable the collection of information associated with the different indicators established to assess judgment criteria and corresponding factors of success.

    A good questionnaire increases the quality of answers. The number, format and order of questions need to be given particular attention to ensure the collection of adequate information and enable the processing of robust analyses. Questionnaires are generally divided into thematic sections, including mostly closed-ended questions (open-ended questions are also possible but will require specific coding to be treated through quantitative analysis). Various techniques exist to collect precise information about respondents’ characteristics, attitudes or preferences (e.g. dichotomous or yes/no questions, multiple-choice questions, mutually exclusive categories, ranking, rating scales, etc.). The questions must be designed as clearly and simply as possible. Pilot-testing the questionnaire before circulating it to the targeted audience is always recommended.

    Conducting a survey or any type of consultation activity requires compliance with data protection rules. Notably, the respondents must be made aware of the data protection policy. Moreover, don’t forget to include a foreword to the questionnaire explaining in simple terms the background and context, as well as the aim and scope of the consultation activity.

  • Step 2 – Administration of the survey:

    • Step 2.1 – Defining the sampling strategy

      The survey is generally administered to a sample of the population. For the information collected to reflect the attributes of the population, the sample needs to be representative. Different surveying sampling approaches can be established: random sampling with the objective of reaching out to a significant number of respondents (the sample needs to be sufficiently large to avoid selection biases), targeted sampling focusing on one category of the population, or stratified sampling reaching out to different sub-categories representative of the population’s overall structure. To capture the effects of CAP interventions, the survey questionnaire must ideally be distributed to beneficiaries and non-beneficiaries in two groups (treated and control groups) with similar attributes.

    • Step 2.2 – Sending out of the questionnaire

      The questionnaire can be distributed to the sample population in different ways (emails, online forms, face-to-face, telephone, etc.). However, some of these approaches require contact information of the sample population. Several strategies also exist to maximise the response rate from the sample population.

  • Step 3 – Analysis of the results: During the analysis, several shortages inherent to the survey method should be considered. Notably, respondents can sometimes provide inaccurate replies for several reasons (misunderstanding, memory gaps, willingness to influence the survey results, etc.). Hence, the survey’s results should be taken with caution. Also, depending on the response rate, some categories of the population can be over-represented, which can increase the risk of selection bias. Therefore, the questionnaire should always entail a first section about the respondent’s profile, asking for its main attributes. This allows the replies collected adequately represent the targeted sample population and ensure the findings' validity.

    Before the analysis, data need to be checked and prepared. It’s important to ensure the consistency of data and cleanse potential duplicates or errors. This might require specific techniques such as integrated control system, cross-checking with other sources, record matching or analytical techniques for the examination of irrelevant data (e.g. unusual amounts above or below pre-determined thresholds).

    To capture the effects of CAP interventions, a survey questionnaire must ideally be distributed to beneficiaries and non-beneficiaries. Different methods exist to compare the two groups and determine to what extent the changes observed can be attributed to the intervention assessed. One consists of comparing changes that occur before and after the introduction of support for beneficiaries and non-beneficiaries, using the difference in difference (DiD) method.

    The replies on respondents’ attributes, preferences and attitudes can be processed through descriptive statistics and simple indicators such as percentages, averages and data distribution. Judgments and opinions collected through rating scales and scores can also be treated quantitatively to enable comparisons between the different groups of respondents. To avoid bias, the survey results must consider the responses by stakeholder group, location, sector, etc.

Main takeaway points

  • Surveys combine qualitative and quantitative approaches for comprehensive data collection.
  • Deductive analysis is used to validate theories with collected data.
  • They fill data gaps by deriving broader conclusions from representative samples.
  • Surveys are vital for collecting primary data on various subjects.
  • A well designed survey is key to obtaining accurate and reliable insights.

Learning from practice

Further reading