Skip to main content

Research Guides

Research Impact & Researcher Identity

Responsible Use of Metrics

Each quantitative metric has its own context and limitations.  It is important that the context and limitations be accounted for when evaluating research or a researcher. 

There is much debate about the use and mis-use of metrics (e.g. Journal Impact Factor, h-index, etc.) for evaluation purposes they were not intended for. Some of the main issues of these discussions are:

Context

  • Quantitative evaluation should support and not replace qualitative expert assessment 
  • Suite of indicators of the value/impact of research/researcher preferred - since any single quantitative metric can be manipulated
  • Different disciplines (e.g. medicine, history) have different publication and citation practices - many metrics are not applicable across disciplines
  • Quantitative metrics cannot be used to infer positive nor negative value judgment
  • Retracted journal articles continue to gather quantitative metrics (e.g. Times Cited)
  • Evaluation processes need to be open, transparent, and communicated so those being evaluated can understand, test and verify the results
  • There can be a time delay (sometimes years) between publishing/communicating a research output and its demonstrable impact 

Limitations

  • Citation databases such as Scopus and Web of Science focus on journal articles - this is only one type of scholarly output
  • Citation databases such as Scopus and Web of Science focus heavily on English language - research is published in many languages
  • Journal Impact Factor (JIF) is a journal-level metric and is not intended to assess an article nor a researcher.  
  • Journal Impact Factor (JIF) should not be used as a sole metric to evaluate a journal; it should be used with informed peer review

What can I do to be responsible about metrics?

  • Understand the context and limitations of metrics
  • Value all research outputs: articles, books, book chapters, data, technical reports, policy papers, code, etc.
  • Encourage responsible authorship practices such as provision of information about specific contributions of each author
  • Cite primary literature in favour of review articles in order to give credit to the author who first reported a finding
  • Engage in open, responsible, ethical and transparent research practices which do not have metrics as the end goal

Key Readings About Responsible Use of Metrics

San Francisco Declaration on Research Assessment (DORA)

  • Developing and promoting best practices in assessing scholarly research
  • The Canadian Association of Research Libraries (CARL) which Includes the University of Toronto Libraries is a signatory to DORA
  • Website: https://sfdora.org/

Leiden Manifesto

Metric Tide