DORA / Responsible research assessment

Developed in 2012, DORA is a set of recommendations that aims to improve the ways that research is evaluated - for example, shifting emphasis away from the venue of publication and towards assessing research on its own merits. A key suggestion is to avoid the use of journal-based metrics such as Journal Impact Factors for assessing individual research and researchers.

You can read the full declaration at https://sfdora.org/read/ and sign it as an individual researcher here.

The background to DORA

Assessing and evaluating the quality and impact of scholarly research is a challenging task.

Journal Impact Factor, a measure originally introduced to help librarians select journals to purchase, is often used as a short-cut to assess the quality of a piece of published research. However, this is problematic for a number of reasons. These reasons are outlined in the DORA Declaration, but include the fact that the data used to calculate Journal Impact Factors are not transparent or publicly available. Using a journal-based metric is also an imprecise and inaccurate way to assess the quality of an individual article.

Other quantitative metrics for evaluating research quality and impact can also be problematic. For example, an emphasis on the number of citations can lead to researchers ‘gaming the system’, and, in turn, can limit the value of the measure. Likewise, a focus on the number of publications produced by a researcher can lead to researchers ‘salami slicing’ a piece of research into multiple ‘minimal publishable units’ rather than one overarching, higher-quality piece. Measuring the impact of a piece of research is also complicated by the fact that a scientific finding’s true relevance is often not clear at the time of publication.

While quantitative metrics can be useful to answer specific questions, it is important to understand exactly what they are measuring and what they are not. Quantitative metrics can be attractive due to their apparent simplicity and authority but can easily be unrepresentative and lead to unwanted effects when improperly applied. When a research output or researcher is reduced to a single value, information is lost - academic research is rich and researchers are multifaceted. 

For this reason, quantitative measures of research impact or quality must be given careful consideration and should only be used in context, combined with other sources such as qualitative input.

DORA emerged as a way of addressing some of these issues. Signatories of DORA commit to avoiding the use of journal-based metrics such as Journal Impact Factors, assessing research on its own merits rather than the perceived merits of the venue of publication, and exploring new indicators of research quality and impact. For institutions, specific commitments of DORA include being explicit about the criteria used to make hiring and promotion decisions, and considering the value and impact of all research outputs (including datasets and software) as well as a broad range of impact measures, including qualitative indicators of research impact, when assessing research. 

You can find more information about responsible use of metrics on the University’s library webpages here.