Kirjaudu Wikiin oikeasta yläkulmasta, jos haluat kommentoida opasta.

|

Log in from the top right corner if you want to comment on the guide.

In recent years, both national and international recommendations have emerged on the evaluation of researcher, emphasising the priority of qualitative evaluation. In line with the principles of responsible researcher evaluation, publication metrics can only be used to support qualitative evaluation.

If indicators of publication metrics are used in the evaluation, it is important to consider the responsibility and appropriateness of the indicators, for example, through the questions presented in the chapter Indicators:

  • Do the selected indicators provide the information needed for the evaluation?
  • Does the indicator take into account the differences between fields of science?
  • Who might be discriminated against by the indicators selected?
  • Is the indicator based on data that is relevant for the subject of evaluation?

Most indicators describe one aspect of a researcher’s or research group’s publishing activities from a specific perspective. It is recommended that several different indicators are used in order to obtain a wide range of information on the publishing activities. On the other hand, the choice of each indicator used must be justified in terms of the needs of the evaluation. In addition to the results of the analysis, background information relevant to the interpretation of the indicators should also be reported.

The main metrics used to evaluate the publication activity of researchers and research groups are based on the number of publications and the number of citations they receive. Publishing activities can also be viewed from the perspective of collaboration and open access: What proportion of the researcher’s or research group’s publications are collaborative or openly available?  Altmetrics can provide information on the visibility of publications on social media and news services.


Identifying the publications of researchers and research groups 

The publication activity of a researcher or research group is examined using a publication list, if available.  Often publication lists are not available, and publications have to be located from existing databases and data sources.  When locating a researcher’s publications in databases, it is important to ensure that the search is as comprehensive as possible and does not include publications by other researchers with the same name.

The identification of researchers and publications in the database can be complicated by name ambiguities: name changes, people with the same name and different spellings of names. Researcher identifiers make it easier to identify researchers and their publications and to track the citations they receive. However, the identifier may not be present in all of the researcher’s publications, and there is always the possibility of errors in the databases: publications may have an incorrect researcher identifier attached to them.  In Finland, it is recommended to use the international ORCID iD. ORCID iD is a persistent, unique digital identifier. Learn more about researcher identification in the chapter Authorship.

The choice of database or data source must take into account the applicability of the database for the analysis in question. International databases do not include researchers’ publications comprehensively, especially in fields where publications are published in languages other than English. Research information systems, on the other hand, contain a wide range of publications in different languages and of different types, but they only focus on the research carried out at a particular university.

Learn more about the citation and publication data sources in the chapters:

Number of publications

When examining the number of publications, it is important to take into account the different publishing practices of the fields of science, for example, in terms of the most commonly used publication types and co-authorship.  In the fields of science where a lot of research is carried out in groups, it may make sense to examine the group rather than the individual researcher.

In addition to the different practices between fields of science, the number of publications is influenced by the resources available, such as the number of researchers working in the research group and the funding received. The number of publications is also affected by the different career paths of researchers, such as parental leave and breaks in their research careers.

Learn more about the number of publications and responsible use of indicators in the chapter Productivity indicators.


Citations

The number of citations received by the publication shows how much it has been used in research since its publication. The citations can be thought to indicate the benefit of the research presented in the publication to the scientific community, and thus its scientific impact. When examining the number of citations, it should be noted that different fields of science have their own publication and citation practices.  The number of citations also depends on the date of publication and the data source used.

The most important sources of citation data are the multidisciplinary Web of Science (WoS), Scopus, Dimensions and Google Scholar. When examining the number of citations, it is worth looking at more than one source of data.

Values with or without self-citations can be taken into account when counting citations and indicators based on citation data. Especially for large research groups with a high output of publications, the number of internal citations can be high.

For more information on citations, see chapters the Citations and Citation impact indicators.


Citation impact indicators

When examining researchers and research groups, it is possible to use the indicators described in the chapter Citation impact indicators.

When using the citation impact indicators, a sufficient citation time window and a sufficiently large number of publications must be taken into account.  It is not necessarily meaningful to calculate indicators for very small numbers of publications and citations.  Indicators calculated from small numbers of publications and citations are also more sensitive to various disturbances, such as deficiencies in data coverage. The chapter Methodological points of view provides more information on the citation time window and the indicators’ susceptibility to interference.


The h-index is a very widely used and well-known indicator. The h-index aims to measure a researcher’s productivity (number of publications) and the impact of their publications (number of citations).  However, there are many issues with and limitations to its use.  Read about responsible use of the h-index and citation impact indicators in the chapters Indicators and H-index.

Research collaboration and networks

One approach to examining a researcher’s or research group’s publishing activities is to look at the research collaboration that has taken place: How many publications have been produced through international or national collaboration or collaboration within the organisation?  When it comes to international collaboration, differences between fields of science must be taken into account.

Research collaboration can also be examined from the perspective of organisation types. This might include examining, for example, how many publications have been produced in collaboration with companies.

Learn more about cooperation indicators and the responsible use of them in the chapter Collaboration indicators.


Indicators used to evaluate journals

The Publication Forum (JUFO) classification is a publication channel classification system to support the quality assessment of scientific publishing activities. The Publication Forum classification was originally intended as a tool for assessing the average quality of large volumes of publications produced by universities. The classification is not intended for assessing the quality of smaller volumes of publications, nor for evaluating or comparing research groups or individual researchers. Learn more about the use of the Publication Forum in the chapter Publication Forum.

The most well-known journal evaluation tool worldwide is the Journal Impact Factor (IF). In line with responsible research evaluation, Impact Factors should not be used to evaluate an individual researcher or a research group, either.  Learn more about the evaluation of scientific journals and the indicators used to evaluate journals in the chapter Most commonly used tools for evaluating publication channels.

Indicators of open access

Open access to a researcher’s or research group’s publications can be examined using different sources of information. Most international databases include information on the open acces of publications. Information on the open access of publications is often also available in organisations’ own research information systems. Databases also contain information on the open access type of publications. These types can be gold OA, bronze OA or green OA, for example. The definitions of open access used by different sources may differ a great deal.

Learn more about the open access publications in the following chapters:

Open access to publications

Indicators of open publishing

Top lists

There are also various top lists of researchers, mostly based on the number of citations received by the publications. From the perspective of responsible publication metrics, researcher lists are problematic because they only value researchers based on the citations received by publications and the indicators derived from them.

Usually, only the most productive and highly cited researchers in those fields of science that have a high coverage in the source used to create the top list are included in the top lists.  Other problems with researcher listings include reliable identification of a researcher’s publications and problems with classifications of fields of science. An example of a top list of researchers is the Highly Cited Researchers list. This list includes only the most cited researchers in the various fields of science from the publications in the Web of Science database.

References

Hirsch, J.E. (2005) An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences 102, 16569–16572.  Available: https://doi.org/10.1073/pnas.0507655102

Responsible analysis of researchers and research groups

The evaluation of researchers should be carried out in accordance with the principles of responsible evaluation. In Finland, two national recommendations for evaluation have been published:

  • Good practice in researcher evaluation. Recommendation for the responsible evaluation of a researcher in Finland. The main principles of the recommendation are transparency, integrity, fairness, competence and diversity.


  • National recommendation on the responsible use of publication metrics. The main principles of the recommendation are:

    1) Publication metrics can be used to support qualitative expert evaluation.

    2) The methods and indicators chosen must be applicable to the subject and purpose of the evaluation.     

    3) The whole evaluation process, including the data, the analytics methods and the results, must be as open and transparent as possible.

Learn more: Finland’s national recommendations.

In addition, several Finnish universities are committed to the DORA declaration, whose key principle is the prioritisation of qualitative evaluation and the avoidance of using metrics based on publication channels as a substitute for measuring the quality of an individual article.

Learn more: International recommendations.

  • No labels