Each quantitative metric has its own context and limitations. It is important that the context and limitations be accounted for when evaluating research or a researcher.
There is much debate about the use and mis-use of metrics (e.g. Journal Impact Factor, h-index, etc.) for evaluation purposes they were not intended for. Some of the main issues of these discussions are:
Context
- Quantitative evaluation should support and not replace qualitative expert assessment
- Suite of indicators of the value/impact of research/researcher preferred - since any single quantitative metric can be manipulated
- Different disciplines (e.g. medicine, history) have different publication and citation practices - many metrics are not applicable across disciplines
- Quantitative metrics cannot be used to infer positive nor negative value judgment
- Retracted journal articles continue to gather quantitative metrics (e.g. Times Cited)
- Evaluation processes need to be open, transparent, and communicated so those being evaluated can understand, test and verify the results
- There can be a time delay (sometimes years) between publishing/communicating a research output and its demonstrable impact
Limitations
- Citation databases such as Scopus and Web of Science focus on journal articles - this is only one type of scholarly output
- Citation databases such as Scopus and Web of Science focus heavily on English language - research is published in many languages
- Journal Impact Factor (JIF) is a journal-level metric and is not intended to assess an article nor a researcher.
- Journal Impact Factor (JIF) should not be used as a sole metric to evaluate a journal; it should be used with informed peer review
What can I do to be responsible about metrics?
- Understand the context and limitations of metrics
- Value all research outputs: articles, books, book chapters, data, technical reports, policy papers, code, etc.
- Encourage responsible authorship practices such as provision of information about specific contributions of each author
- Cite primary literature in favour of review articles in order to give credit to the author who first reported a finding
- Engage in open, responsible, ethical and transparent research practices which do not have metrics as the end goal
