Citation impact indicators

Indicators of scientific impact are mainly based on the number of citations received by publications. The number of citations received by publications, the average number of citations for a set of publications and the h-index are examples of citation impact indicators. Disciplinary differences regarding citation practices, age of publications and publication types have been taken into account in the normalised citation impact indicators.

The number of citations

The number of citations received by the publication shows how much it has been used in research since its publication. The citations can be thought to indicate the benefit of the research presented in the publication to the research community, and thus its impact. However, impact is a broad concept, and it can be interpreted in many different ways, so we will use the term citation impact in this guide. Learn more about the number of citations as an impact indicator in the chapter Citations.

Responsible use

• Consider the different publishing and citation practices of different fields of science as well as the publications’ publishing date.
• The number of citations is database dependent. The coverage of publications from different disciplines in databases may vary greatly.
• See also the questions related to responsible evaluation from the page Indicators.

Mean number of citations

The mean number of citations can be calculated for different publication sets. The mean values are utilised, for example, in normalised indicators. However, for skewed citation distributions, mean values may not always be illustrative indicators.

Responsible use

• Consider the different publishing and citation practices of different fields of science as well as the publication date.
• The number of citations is database dependent. The coverage of publications from different disciplines in databases may vary greatly.
• Review how well the mean value describes your material. Means often do not depict citation distributions as the distributions tend to be highly skewed. There can be great differences between mean and median values.
• Individual publications with a high number of citations can have a major impact on the mean, and are quite common.
• See also the questions related to responsible evaluation from the page Indicators.

Normalised indicators

Normalised indicators refer to indicators that take into account the publication’s field of science, age and type. Determining the publication’s field is a key step in normalisation. In databases, publications are classified under different scientific disciplines based on various grounds and methods. Many international databases use classifications based on field of science classification of publication series: a publication (e.g. a scientific article) is categorized to the same subject category as the publication series (e.g. a journal) in which it has been published. More information about the classifications is available in the chapter Differences between fields of science. 

Publications are grouped in terms of their age, discipline/field and publication type. A citation distribution for each group is formed by ordering publications based on their received citations. A mean value of citations is calculated for each distribution. Normalisation is based on the distibutions and mean values. The use of field-normalised indicators aims to allow for comparisons of publications accross disciplines. The sections below present these various field-normalised indicators.

Field-normalised Top x% indicators

The field-normalised Top x% indicators are based on the database’s field-specific citation distributions. The Top x% indicators show the proportion of the analyzed publication set that belongs to the most frequently cited x percentile of the publications of the same field, publication year and publication type. Top x% indicators are used as indicators of high citation impact. For example, Top 10% share and Top 1% share demonstrate the relative share of publications reaching both high (10%) and top (1%) impact within the publication set.

Top x% -indices

The Top x% -index show the relative proportion of the analyzed publication set that belongs to the most frequently cited x percentile of the publications of the same field, publication year and publication type. For example, a Top 1% index value exceeding 1 means that more than 1% of the publications in the analysed publication set belong to the most cited 1% of the field of science in question.

Example: how to calculate the Top 10% index 

• Calculate the citation distribution: In 2014, 25,000 publications were published in the field of science Y. This means that the Top 10% of the citation distribution includes 2,500 publications.
  
• The analysed set of publications: Faculty Z works within the same field of science. The researchers of this faculty published 500 publications in 2014. Of these publications, 100 are within the Top 10% of the aforementioned citation distribution. Statistically, the expected number of the faculty’s publications included in the top 10% is 50 publications, i.e. 10% of the faculty’s publications.
  
• Calculating the Top 10% index: The Top 10% index is calculated by dividing the number of the faculty’s publications within the top 10% by the expected value, i.e. 100/50 = 2.
  
• Please note that this is a simplified example; in reality, as the publications belong to different fields of science and have been published in different years. Thus, the analysed publications will be compared to many different citation distributions. Additionally, the Top x% class rarely includes an exact x% of the publications, meaning that the Top 10% class hardly ever includes exactly 10% of the publications. This is due to the fact that the citation number is a discrete variable that is only assigned integer values.

Top x% shares

The Top x% share shows the percentage of the analysed publications belonging to the most frequently cited x percentile of the same field of science, publication year and publication type. For example, a value 2% for the Top 1% share means that 2% of the publications in the analysed publication set belong in the most cited 1% of the field of science in question. 

Example: how to calculate the Top 10% share

• Calculate the citation distribution: In 2014, 25,000 publications were published in the field of science Y. This means that the Top 10% of the citation distribution includes 2,500 publications.
• The analysed set of publications: Faculty Z works within the same field of science. The researchers of this faculty published 500 publications in 2014. Of these publications, 100 are within the Top 10% of the aforementioned citation distribution.
• Calculating the Top 10% share: calculate the percentage of the faculty’s publications belonging to the Top 10% of all the faculty’s publications: 100/500 * 100%=20%
• Please note that this is a simplified example; in reality, the publications belong to different fields of science and have been published in different years. Thus, the analysed publications will be compared to many different citation distributions. Additionally, the Top x% class rarely includes an exact x% of the publications, meaning that the Top 10% class hardly ever includes exactly 10% of the publications. This is due to the fact that the citation number is a discrete variable that is only assigned integer values.

Top x% publications

The Highly Cited Papers and Hot Papers of the Essential Science Indicator and Web of Science database are one version of the Top x% indicators. They are based on the publications’ field-specific citation distributions. The Highly Cited Papers and Hot Papers statuses are assigned to publications that exceed the limit values of citations determined in the database. 

The Highly Cited Papers status is granted to publications no older than ten years old that exceed the field-normalised limit value of citations required to belong to the Top 1% of most cited papers.

The Hot Papers status is granted to publications no older than two years old that exceed field-normalised limit value of citations required to belong to the Top 0.1% of most cited papers that were added to the database during the same update. The updates are done every two months. For Hot Papers, when counting citations, only the citations by publications that were added in the latest update will be counted.

Responsible use

The Top x% indicators seek to account for disciplinary differences regarding publication and citation practices as well as accounting for the publication date. They are not sensitive to individual publications with a high number of citations.

Remember that

• Top x% indicators only assign value to a small proportion of the publications, while others are ignored.
• Drawing the limit at a certain percentile is arbitrary; instead of Top 1% and Top 10% we could just as well analyse the Top 5% and Top 15%.
• The indicators are based on classifications of fields of science that are not entirely unproblematic (more information on this can be found in the chapter Classification of fields of science.
• When reviewing the Top x% indicators of fairly recent publications (published less than five years ago), it should be noted that the citation numbers are low in certain fields of science. In such cases, a meaningful distribution for calculating the Top x% indicator does not yet exist. 
• Different types of publications accumulate citations in different ways. A conference publication, for example, can become included in the Top x% publications with far fewer citations than a journal article.
• See also the questions related to assessing responsibility in the chapter Indicators.

Field-normalised citation indicators

Field-normalised citation indicators reflect the citation impact of the publications in their respective scientific fields. The use of field-normalised indicators aims to allow for comparisons of citation impact of publications accross disciplines.

These indicators describe the relationship between the number of citations received by a publication or a set of publications and the mean value of citations received by the same type of publications that were published within the same scientific field in the same year. The value corresponding the average citation impact is 1. Values below 1 indicate a lower than average number of citations while values above 1.0 indicate a number of citations that is higher than average. When a unit is assigned score of 1.78, for example, this means that it has 78% higher number of citations than the global average.

Examples: how to calculate field-normalised citation indicators.

Example 1. A review article has been cited 15 times. Review articles in journals of the same field that have been published in the same year have received an average of 12 citations. Calculating the normalised citation index of the review article: 15/12= 1.25.

Example 2. When calculating a normalised citation index for a set of publications, the citation indices of individual publications need to be combined. Calculating a field-normalised citation index for a set of three publications:

Responsible use

The field-normalised citation indicators seek to account for the differences between fields of science regarding publication and citation practices as well as accounting for the publication date.

Remember that

• The indicators are based on mean values. Individual publications with a high number of citations can have a major impact on the indicator’s value. 
• When reviewing fairly recent publications, it should be taken into account that the citation numbers are lower within certain fields, due to which small changes in the number of citations can lead to major changes in the indicator value.
• The indicators are based on classifications of fields of science that are not entirely unproblematic (more information on this in chapter Classification of fields of science.
• See also the questions related to assessing responsibility in the chapter Indicators.

Normalised indicators in databases and other sources

H-index

One of the best known bibliometric indicators is the h-index, also known as Hirsch index. It was developed by Jorge E. Hirsch in 2005. Read more about the topic in the original article. 

A researcher’s h-index is determined as follows: the researcher’s h-index value is h, if h number of their publications have received at least h citations and rest of their publications have received a maximum of h citations.

Example.
An h-index value of 10 means that a researcher has 10 publications that have all at least 10 citations. Their potential other publications have been cited a maximum of ten times.

Example.
The h-index value of two researchers can be the same even if their publication profiles (number of publications and the number of citations received by them) are very different.

• Researcher 1 has published five articles that have been cited the following times, respectively: 9, 9, 7, 6, 5 
• Researcher 2 has published 11 articles that have been cited the following times, respectively: 200, 150, 99, 70, 5, 4, 4, 3, 2, 1, 1 
• Both researchers have an h-index value of 5.

Responsible use

• Consider the different publishing and citation practices of different fields of science as well as the publication date.
• The number of citations is database dependent. The coverage of publications from different disciplines in databases may vary greatly.
• The h-index cannot be greater than the number of publications. For example, if the researcher has published 10 publications, the maximum value is of h-index is 10 even if the publications of the researcher in question have been cited hundreds of times.
• The h-index is a cumulative indicator and cannot decrease. It can also be determined for a shorter review period (such as for five years) instead of the researcher’s entire career. 
• The h-index tends to favour disciplines with larger research groups and hence greater number of authors per publication. 
• The h-index does not take into account the authors’ roles in the publication. A key role in the actual writing process is counted the same as any other role.
• The length of a researcher’s career impacts their h-index. A researcher who has been publishing works for a long time is more likely to have a high h-index than a researcher who is at the early stages of their career. The more experienced researcher has had a longer time to publish work and more time to accumulate citations. 
• The h-index does not take into account when the author published his/her work or whether they have stopped publishing altogether. The h-index may continue to grow even after the researcher has stopped publishing as their publications continue to be cited.
• The h-index does not highlight a researcher’s individual publications with a high impact.
• See also the questions related to assessing responsibility in the chapter Indicators.

H-index in data sources