“The current system for assessing research, researchers and research institutions does not incentivise or reward enough the quality of all research outputs in their diversity. It often relies on the quantity of publications in journals with high Journal Impact Factor and citations as mere proxies for quality and impact, thereby underestimating the value of other contributions, lowering reproducibility, and holding back researchers from open sharing and collaboration.”
(Paris Call on Research Assessment, 2022)

The evaluation and review of research and scholarship is a central activity of research-performing organisations, scientific institutions, and research funding organisations, as well as researchers. Numerous high-level stakeholders, including UNESCO, the European Commission, SCIENCE EUROPE, LERU, and EUA, have recently called for a reflection on current evaluation practices, and have published recommendations, declarations, position papers, statements, manifestos, and calls proposing concrete agendas for reform. Although there is broad consensus regarding the need for fundamental reform, defining the specific indicators and criteria to be used, along with the question of how they should be weighted, remains fraught with controversy. Will including such criteria as Open Science, diversity, and others perhaps undermine the focus on excellence?

In our panel discussion, representatives of different research disciplines and organisations explored which criteria could help to evaluate research in the future, and how these criteria might take into account disciplinary differences.
 

Guiding questions included:

• How can we ensure that excellence remains the key criterion guiding research funding decisions?
• How can current institutional practices in academia be aligned with the international quest for opening up science and establishing responsible use of research metrics as part of a holistic evaluation, where different methods and approaches complement each other?
• What are the disciplinary and local differences in research assessment?
• How can the different stakeholders (in research production, research policy and administration, research consumption, and use and evaluation services) better collaborate to implement good evaluation practice (transparency, integrity, competence, diversity)?
• How can we avoid (overcome) a ‘double bind’ (in particular among early career researchers) while old and new assessment principles coexist?
• Which structural and resource problems in academia interfere with the establishment of a novel assessment culture?
• Which unintended consequences need to be considered when altering the current system?
• How can we establish an evidence base for the efficacy, efficiency, and fairness of novel assessment practices?