Workshop: "Semantic analysis for innovation policy"

12-13 March 2018

Paris - OECD Conference Centre (room CC20)

 

 

The digital transformation offers new tools for innovation policy analysis. With software developments and increased computing capacities, the possibility to systematically exploit textual information, as illustrated by the analysis of the OECD TIP working party reports that were conducted in the context of the TIP@50 event, has grown enormously. The types of data that can be usefully analysed to support STI policy are broad, ranging from official policy papers and evaluations, to administrative data held in government databases, to social media. Such analysis can help identify emerging trends and monitor policy progress over time. However, while semantic analysis holds enormous promise, careful data preparation and exploitation are needed, otherwise results may be misleading. The objective of the workshop was to shed further light on the potential of semantic analysis, highlighting the caveats and challenges on the way. The workshop also focused on the specific semantic analysis that was conducted of TIP documents. 

 

AGENDA (PDF) 

SUMMARY OF DISCUSSIONS

 

Shortcut URL: www.innovationpolicyplatform.org/semantics

 

Organisation

The workshop was a joint workshop of the CSTP (the DSIP and REITER projects), and the TIP working party, relating to its ongoing projects on open and digital innovation and on assessing the impacts of knowledge transfer and policy.

The workshop was organised back-to-back with the CSTP-TIP workshop on 14 March that explored new models and business perspectives on knowledge transfer between industry and science.

There was also a workshop on 13 March organised by the CSTP and GSF on open data.

The workshop was organised around two related but separate topics:

DAY 1: Semantic analysis of TIP documents & best practice in semantic analysis (12 March)

DAY 2: Semantic analysis to investigate knowledge transfer and innovation policy (13 March)

Objectives of the workshop

 

  • Explore the potential of semantic analysis to exploit qualitative information for innovation policy analysis on knowledge transfer between industry and science in particular
  • Learn about existing national initiatives that semantically exploit large datasets to inform policy making and analysis
  • Get a hands on experience of the possibilities of text-mining (and its limitations)
  • Learn more about the evolution of innovation policy from text-mining of the TIP corpus

Agenda

DAY 1: Monday, 12 March 2018 - Semantic analysis of TIP documents & best practice in semantic analysis

9h00 - 9h15: Coffee

9h15-9h30: Welcoming and introduction to the workshop

  • Dominique Guellec, Head of Division, Directorate for Science, Technology and Innovation, OECD
  • Michael Keenan (Senior Policy Analyst, OECD) and Caroline Paunov (Senior Economist, OECD) 

9h30-11h00: Session 1 - Potential, best practice and caveats in using semantic analysis

The session introduced the basics of semantic analysis (methods, software tools, principles and challenges) to establish a common ground for discussion between experts and non-experts alike and focused notably on the following issues:

  • What can be done using semantics analysis in the field of innovation policy? What are some illustrative examples? 
  • What are the strengths and caveats involved in such an analysis compared to more traditional types of analyses? 
  • What are the steps that need to be undertaken to undertake such an analysis? 
  • What applications of semantic analysis relevant to innovation policy can be implemented as of today? What applications need further development? 

Chair: Dominique Guellec, Head of Division, Directorate for Science, Technology and Innovation, OECD

Speakers :

Roundtable discussion involving Mary-Ann Grosset, Digital Practice Team Manager at OECD; Philippe Laredo, Research Director at Université Paris-Est and Professor at Manchester University; Juan Mateos, Head of Innovation Mapping, NESTA, United Kingdom; and Margherita Russo, Professor at University of Modena and Reggio Emilia, Italy. 

11h00 - 11h30: Coffee break

11h30-13h00: Session 2 - In-depth semantic analysis of TIP documents

The session allowed for an in-depth discussion of the semantic analysis that was conducted in support of the TIP@50 event and focused in particular on the following questions:

  • How has innovation policy thinking changed in what is reflected in 25 years of TIP, CSTP and national policy discussions?
  • What trends can be identified over the 25-year period for the TIP, CSTP and beyond? 
  • What lessons can be taken from the work for the future of the TIP, the CSTP and innovation policy analysis more generally? 

 

Chair: Caroline Paunov, Senior Economist, Directorate for Science, Technology and Innovation, OECD

Speakers : 

  • Michael Keenan, Senior Policy Analyst, Directorate for Science, Technology and Innovation, OECD
  • Margherita Russo, Professor, University of Modena and Reggio Emilia, Italy; and Pasquale Pavone, Researcher, Research Centre for the Analysis of Public Policies (CAPP), University of Modena and Reggio Emilia, Italy
  • Dirk Meissner, Professor,  National Research University - Higher School of Economics, Russian Federation
  • Philippe Laredo, Research Director at Université Paris-Est and Professor at Manchester University; and Antoine Schoen, Professor, ESIEE Paris, France
  • Byeongwon Park, Research Fellow, Science and Technology Policy Institute (STEPI), Korea

13h00 - 14h30: Lunch break

14h30 - 16h00: Session 3 - The making of the TIP@50 analyses

The session focused on what lays behind producing the TIP@50 analyses and offered interested participants an opportunity to do a simple analysis:

  • What material was used and how did the choices of how the material was organised and analysed affect the analysis?
  • What software decisions were made? What are the costs involved in using such software? What is freely available and what is proprietary?
  • How can these analyses be applied to other types of analysis of innovation policy?  

Chair: Michael Keenan, Senior Policy Analyst, Directorate for Science, Technology and Innovation, OECD

Speakers : 

  • Jan-Anno Schuur, Information Systems Specialist, OECD; Andrés Barreneche, Policy Analyst, OECD
  • Margherita Russo, Professor, University of Modena and Reggio Emilia, Italy; and Pasquale Pavone, Researcher, Research Centre for the Analysis of Public Policies (CAPP), University of Modena and Reggio Emilia, Italy
  • Dirk Meissner, Professor,  National Research University - Higher School of Economics, Russian Federation
  • Philippe Laredo, Research Director at Université Paris-Est and Professor at Manchester University; and Antoine Schoen, Professor, ESIEE Paris, France

16h00 - 16h30: Coffee break

16h30-17h00: Session 4: Perspectives on semantic analysis

Speaker : 

  • David Chavalarias, Research Director at the National Centre for Scientific Research (CNRS) and Director of the Complex Systems Institute of Paris Ile-de-France (ISC-PIF), France: Reconstruction and monitoring of the scientific debates from digital traces: the cases of science and politics

17h00 - 18h30: Session 5 - Hands-on exercise

The session allowed participants to produce a simple semantic analysis with the support of those involved in the TIP@50 analysis. This required bringing a laptop. 

  • Antoine Schoen: Replicating results from TIP@50 analysis using Cortext (Find the instructions to access the Cortext Platform here and access the TIP corpus here). Find the instructions of the exercise here.

DAY 2: Tuesday, 13 March 2018 - Semantic analysis to investigate knowledge transfer and innovation policy

9h00 - 9h15: Coffee

9h15 - 9h45: Brief summary of main conclusions of 13 March regarding semantic analysis

9h45 - 11h15: Session 6a - Examples of semantic analysis conducted to study knowledge transfer

This session presented different national initiatives that use semantic analysis to shed light on innovation policy and in particular on knowledge transfer between industry and science.

  • What new questions can be addressed using semantic analysis?
  • What information are they semantically analysing and how is this accessed?
  • What infrastructure requirements were needed to set them up?
  • What new answers have these studies identified, with a special focus on knowledge transfer? 

 

Chair: Frédérique Sachwald, Director, Science and Technology Observatory (OST), HCERES, France 

Speakers: 

  • Antoine Schoen, Professor, ESIEE Paris, France: KNOWMAK
  • Sam Arts, Assistant Professor, KU Leuven, Belgium
  • Cinzia Daraio, Associate Professor, University of Rome La Sapienza, Italy 
  • Francesco Osborne, Research Fellow, Knowledge Media Institute, The Open University, UK

11h15 - 11h45: Coffee break

11h45 - 13h00: Session 6b - Examples of semantic analysis conducted to study knowledge transfer

Chair: András Hlács, Counsellor, Permanent Delegation of Hungary to the OECD

Speakers: 

  • Andrés Barreneche, Policy Analyst, Directorate for Science, Technology and Innovation, OECD
  • Juan Mateos, Head of Innovation Mapping, NESTA, United Kingdom: Mapping and strengthening research and innovation networks with open data in Arloesiadur (see background reading paper)
  • Maarten van Leeuwen, Lecturer, Leiden University, the Netherlands [by WebEx]
  • Georg Licht, Head of Department of Economics of Innovation and Industrial Dynamics, Centre for European Economic Research (ZEW), Germany

13h00 - 14h30: Lunch break

14h30 - 16h00: Session 7 - Hands-on experience

Parallel sessions: 

Session A: Using semantic analysis tools to analyse data from the EC/OECD 2017 STI Policy Survey (room CC20)

Presenter: Andrés Barreneche, Policy Analyst, OECD

Description: The OECD Secretariat has recently revised the data collection methodology of the STI Policy Survey, run jointly with the European Commission. Data provided by countries is now more firmly structured on taxonomies-ontologies, which not only improves the comparability of responses but also facilitates analysis. Visualisation tools, accessed through a web interface, take advantage of semantic structures to aggregate data and present insights on over 6000 policies initiatives submitted by over 50 countries. This hands-on exercise provided participants the opportunity to get familiarised with these tools and learn how they can readily use the database for their own purposes.

Session B: From text to impact in ninety minutes (room MB3122)

Presenters: Juan Mateos (Head of Innovation Mapping, NESTA), Chantale Tippett (Principal Researcher, Innovation Mapping, NESTA), and Joel Klinger (Data Developer, Innovation Mapping, NESTA)

Description: This session gave participants an opportunity to turn their text into insight by walking them through key phases of the project pipeline. Starting with an overview of how semantic analyses can be used to explore questions of interest in the domain of knowledge transfer, the session was broken down into a series of practical exercises covering data collection, analysis and outputs. These consisted of a combination of presentations drawn from Nesta’s vast experience and small group exercises. The practical component concluded with a short session on the practicalities of semantic analyses, allowing participants to reflect on how they might implement projects in their own institutional context. The session concluded with a Q&A on the semantic analysis pipeline and a recap of key points. Find the instructions for this session here and the databases for the exercise here.

Session C: Supporting research policy makers with semantic technologies (room MB4122)

Presenter: Francesco OsborneResearch Fellow, Knowledge Media Institute, The Open University, UK

Description: The number of papers and the available scientific knowledge is growing rapidly, making it harder to keep track of all the relevant knowledge that could inform research policy makers. This session discussed how semantic technologies, that are being increasingly used to represent and analyse research data, can help to tackle this issue. It addressed some novel solutions for predicting trends, collecting research materials, analysing the research landscape, and tracking the evolution of technologies. The session also included a hand-on exercise in which the attendees were given access to the demos of some of these prototypical systems. More information about the technologies showcased during the session are available at: http://skm.kmi.open.ac.uk/#projects 

16h00 - 16h30: Coffee break

16h30 - 17h50: Session 8 - Concluding panel and next steps

  • What do you think is the potential and feasibility for semantic analysis in your own context?
  • What could be useful next steps to facilitate exchange on best practice?
  • How can CSTP and TIP provide support to help exploit the potential of these tools?
  • What other tools should be part of the basket to consider? 

 

Chair: Göran Marklund, Deputy Director General for External Matters, VINNOVA, Sweden, and Chair of the TIP Working Party

Speakers:

  • Marnix Surgeon, Deputy Head of Unit, European Commission
  • Frédérique Sachwald, Director, Science and Technology Observatory (OST), HCERES, France 
  • Armin Mahr, Head of STI locations and Regional Policies, Federal Ministry of Science, Research and Economy (BMWFW), Austria 
  • Cinzia Daraio, Associate Professor, University of Rome La Sapienza, Italy 

17h50 - 18h00: Wrap up

Michael Keenan and Caroline Paunov, OECD