A recent working paper presents the challenges in studying the impact of large-scale research and innovation funding programmes.
The European Union has consistently shown a strong commitment to supporting research and innovation (R&I) through its successive Framework Programmes. Horizon Europe, with a budget of €95.5 billion, stands as the largest and most ambitious of these programmes. Alongside it, a range of other competitive and multi-actor EU funding instruments contribute to shaping a more innovative, resilient, and future-oriented Europe. A recently published working paper by the Directorate-General for Research and Innovation addresses the complexities and methodological challenges involved in assessing the real-world impact of such large-scale R&I programmes.
Research and innovation play a vital role in securing Europe’s future, offering tools to address a wide array of social, environmental, economic, and political challenges. The EU’s framework programmes are central to these efforts, funding a broad spectrum of R&I activities and encouraging international cooperation, for instance, through Switzerland’s participation in several EU programmes. These initiatives not only drive scientific progress but also strengthen collaboration across borders and sectors.
Evaluating the impact of these funding instruments is both complex and essential. Empirical evidence helps quantify benefits, improve programme design, and ensure that future investments are more targeted and effective. A robust evaluation process also gives policymakers the insights needed to identify strengths and to address gaps. The working paper outlines a central question for such evaluations: ‘Does receiving a grant under a European Framework Programme have an impact on the achievement of key policy objectives?’ Impact can take various forms, including (1) advancing fundamental human knowledge; (2) driving productivity growth that translates into economic growth; (3) facilitating scientific progress to tackle social and environmental challenges; and (4) promoting equity in access to education or technology and enhancing social mobility.
The paper highlights several major challenges in measuring the effects of EU R&I funding, such as estimating long-term outcomes and capturing spillover effects. One of the core goals of R&I funding is to promote knowledge spillovers, something actively encouraged by Horizon Europe through its emphasis on cross-border collaboration and open science. These spillovers often extend beyond the direct beneficiaries of funding, complicating efforts to determine whether outcomes result from the funding itself or from indirect effects through collaboration within research networks and peer environments.
Estimating long-term impact presents another major challenge. Many of the objectives pursued by R&I programmes require time to materialise and are often difficult to measure. This is particularly true for initiatives under Horizon Europe that focus on fundamental research or address long-term systemic issues like climate change. Standard policy evaluations often rely on short-term indicators, which may not fully capture the deeper and more gradual effects of such investments.
Another difficulty stems from the wide thematic scope of programmes like Horizon Europe, which span numerous disciplines and sectors. In contrast, more narrowly focused programmes such as EU4Health have defined thematic priorities. These differences lead to varied results and spillovers across sectors and regions, making it harder to isolate and compare the specific impact of each instrument. As a result, direct comparisons, whether across different programmes or even between different calls within the same programme, must be made carefully and with attention to context.
In conclusion, the working paper points out that evaluating R&I funding programmes brings both valuable opportunities and certain methodological challenges. Since the impacts can be felt at different levels, from individual beneficiaries to wider regional outcomes, it is important to approach the evaluation process with care. One key issue is whether the findings from assessing a specific instrument can be generalised and applied to other areas within the Horizon Europe programme. Because the programme covers a broad range of thematic focuses, direct comparisons are not always possible. Thus, evaluators need to be cautious and use approaches that take the diversity of the EU funding landscape into account.