Publication Forum manual for panels

1 Introduction

Publication Forum manual for panels contains information on the Publication Forum, scientific publications and publication channels, classification criteria, principles of the evaluation, and panels' work. 

2 General

2.1 Publication Forum

The Publication Forum established in 2010 under the auspices of the Federation of Finnish Learned Societies (TSV) is a system to support the quality assessment of scientific publication activities in all disciplines, based on the classification of domestic and foreign publication series and book publishers. The rating decisions are taken by the over 200 expert members of the 23 Evaluation Panels covering various disciplines, universities and research institutes. The Panel members are nominated on basis of proposals made by the universities, government research institutes, learned societies and science academies.

2.2 Steering-group

The Panels are appointed and the evaluation work supervised by the Publication Forum Steering Group, nominated by the TSV Board and constituted by representatives of the Ministry of Education and Culture (MinEdu), the Finnish Council of University Rectors (UNIFI), the Academy of Finland, the Council for Finnish University Libraries (SYN), the National Library of Finland, the IT Centre for Science (CSC) as well as various major disciplines. The work of the Steering Group and the Evaluation Panels is prepared and supported by the Publication Forum Secretariat employed by TSV which also provides the Panels with meeting facilities at the House of Science and Letters in Helsinki. The IT services required by the Publication Forum are run in collaboration with CSC.

2.3 Levels

In the publication forum, the scientific publishing channels are divided into three categories: basic scientific channels (level 1), leading scientific channels (level 2) and top scientific channels (level 3). In addition, all publishing channels evaluated by panels that do not meet the minimum criteria for Level 1 are listed in Category 0. Category 0 does not give the publication channel any special status, but only indicates that the channel did not meet the criteria for Level 1 at the time of assessment.

2.4 Publication Forum functions

The publication forum rating serves as an indicator of the quality of university publication production in the ministry of education and culture’s funding model from 2015 onwards.Combined with publication data collected by MinEdu, the classification provides a comprehensive and comparable information on the volume and development of the publishing activities of organizations and disciplines. The classification disseminates information on the quality, impact and prestige of scientific journals, series and book publishers in the scientific community, and thus encourages Finnish researchers to publish their most important research in the leading national and international scientific publications channels in their field.

2.5 Use of Publication Forum classification in evaluations

In the internal use of research organisations, the classification is best suited for the follow-up of the progress made within the research area’s or unit’s own publication operations. The results can be used as information to the expert panels evaluating their research. Using the classification mechanistically to allocate financing to research units may lead to inequitable treatment of different fields of science. Due to the differences in publication practices and research questions and methods, the number of publications on the higher level publication channels produced by individual researchers varies both between and inside the disciplines and fields of research. The publication channels of Level 1 may contain individual publications that are above the average in terms of quality and impact, while the publication channels classified as Level 2 and 3 may include publications that do not meet the average quality and may be never cited. For these reasons, the classification cannot substitute for peer evaluation as a criterion or grounds for decisions on an individual researcher's recruitment, financing or rewarding.

3 Scientific publications and publication channels

3.1 Scientific publication

‘Publication’ refers to a written presentation of the research outcome by an individual researcher or group of researchers, published as an article in a journal, congress proceedings, as a part of a compilation or an independent monograph, depending on the discipline. To meet the criteria of ‘scientific’, the publication must present new research outcome in a form to be accessible, repeatable and usable by the scientific community. The main target group of a scientific publication is constituted by the experts in the discipline. In the publication type classification used by the ministry of education and culture, scientific publications correspond to types A and C.

3.2 Scientific publication channel

A ‘scientific publication channel’ refers to printed and digital publication series as well as to book publishers specialised in the publication of scientific research outcomes: they have an editorial board constituted by experts, and the publication calls for quality evaluation recognised by the scientific community, above all referee/peer evaluation. Scientific publication channels are evaluated primarily in terms of their scientific impact, not their societal impact. Scientific publication channels may also produce not-peer-reviewed publications, such as comments, reviews and handbooks. In Publication Forum publication channels are divided in three types that are journals/series, conferences and book publishers.

3.2.1 Journals/series

A publication series can be a scientific journal, book series or a series of proceedings of a regularly organised conference. Journals, book series and proceedings with ISSN are classified using the title recorded in the international ISSN centre. The main rule is that the book series of Finnish book publishers are classified separately in the Publication Forum, while the book series of foreign book publishers are not classified.

3.2.2 Conferences

In addition, the classification includes a limited number of conferences listed under the name of the event, the type of channel in this case being "conference". The panel 2 (Computer and information science) and 9 (Electrical and Electronic engineering, information engineering) can classify a conference separately in cases where the level of the main publishing channel, ie the publisher (e.g. IEEE or ACM) or the publication series (e.g. LNCS), does not reflect the true level of the publications of the specific conference. Conferences that have their own publication series with ISSN that only publish proceedings of this conference will be treated as journals/series with the name registered at the ISSN Center (only if the ISSN is actually included in the publications). For conferences that do not have their own (active) ISSN-recognized publishing series, either a publication series with ISSN (in the case of a journal or a book series specializing in proceedings) or an ISBN-based book publisher is primarily identified as a publishing channel.

3.2.3 Book publishers

Book publishers are listed under the name in the ISBN registry or under other established name. In the Publication Forum and the data collection the “book publisher” means a publisher who is responsible for the publisher's ISBN (not other kind of publisher or printing house). Publishers who use self-publishing ISBNs are not primarily treated as book publishers. The imprint of the parent company is classified as an independent publication channel. Publisher’s activities in different countries are considered to belong to the same publication channel. If the faculties or institutes have their own ISBNs, they are considered the same publication channel as the university.

3.3 Special cases

3.3.1 New publication channels

As a rule, publication channels will not be evaluated by the panels if publication activities are less than one year old, or if there are not enough issues or publications in the channel that allow content to be evaluated.

3.3.2 Channels without ISSN or ISBN

Publication channel must have a registered ISSN or ISBN number. The exception is a small number of conferences categorized under an established name that do not have their own ISSN-recognized publishing series.

3.3.3 Professional and popular channels

Publication Forum does not classify channels targeted at professional or general audiences. If the boundary is not clear, channels for the professional and the general public can be evaluated by the panels. If the panels find that such channels are not scientific publication series or book publishers, they will be transferred to a separate list of professional and general publication channels.

3.3.4 Descriptors

Nature and Elsevier have started publishing channels to support open access and reuse of research data. In the articles, the researchers describe the published data, including the methods and technical analyzes used for generating/collecting the data. Journals publishing only data descriptions are placed directly in level 0 without evaluation, but the development of the status of these channels is monitored. Journals publishing only data are normally processed for panel evaluation. Panels need to evaluate carefully whether the journals meet the criteria of level 1 (especially the criteria of specialising in publishing scientific research results, and the criteria of peer review).

3.3.5 Software publishing channels

In computer science publishing channels have been set up for publishing open source software and descriptive articles. Journals specializing in articles describing software are normally processed for panel evaluation.

3.3.6 Short Monographs

International publishers, for example, Palgrave Macillan and Routledge, have launched a new peer-reviewed publication format, the publication type of which corresponds to a monograph but the length of manuscripts is between an article and a traditional monograph. In the MinEdu data collection short monographs are instructed to be classified as book articles. If the publication channel for short monographs can be identified with an ISSN, it can be evaluated in the Publish Forum.

3.3.7 Distribution of publication channels between panels

The journals/series and conferences are divided between the panels according to the discipline, so that the responsibility for evaluating each publication channel is limited to one panel. The exception to this rule is a small number of multidisciplinary journals (e.g. Nature, Science and PNAS). These multidisciplinary publications series, as well as most of the book publishers, have been placed on a panel of general publication channels (panel 24), which is common to all panels. All panels can participate in evaluation of the channels placed in Panel 24.

4 Classification criteria applicable to publication channels

4.1 Publication Forum coverage

The Publication Forum includes active publication channels that specialize in publishing of research results and are relevant from the point of view of Finnish research. The classification thus covers the leading and internationally renowned journals/series and book publishers of the various disciplines as widely as possible, but national and local publication channels in different countries can be limited to those used by or relevant to Finnish researchers.

4.2 Level 1

Level 1 is the basic category of the Publication Forum classification that contains vast majority of foreign and domestic peer-reviewed publication channels. Open access journals are accepted to the classification and evaluated using the same criteria applicable to conventional publication series. Level 1 can be awarded to domestic and foreign journals/series, conferences and book publishers meeting the criteria of a scientific publication channel:

• The channel is specialised in the publication of scientific research outcomes;

• There is an editorial board constituted by experts;

• The scientific publications are subject to a peer evaluation focusing on the entire manuscript.

In principle, a publication channel meeting the criteria of a scientific publication channel must not be included in level 1 if

• over half (1/2) of the reviewers or authors come from the same research organisation that is also the publisher;

• the scientific level or relevance raises questions.

4.2.1 Specialization in the publication of scientific research results

To level 1 can be accepted channels that publish peer-reviewed publications on a regular basis. Channels that occasionally publish individual scientific publications will not be accepted for level 1. Scientific publishing channels may also include not-peer-reviewed publications for both the professionals and the general public. However, it is recommended that the classification channels that are acceptable to the level 1 indicate as clearly as possible which publications are peerreviewed.

4.2.2 An editorial team of science specialists

Level 1 channels must have an editorial board including specialists in the field working in universities or research institutes.

4.2.3 Peer Review

Peer Review refers to a procedure whereby a journal, conference or book publisher invites experts in the field to make an evaluation of the scientific merits and fitness for publication of manuscripts offered for publication. The peer review should focus on the entire manuscript to be published, not just of the abstract.

4.2.4 Local channels

Only such channels can be approved for Level 1, in which researchers from different research organizations publish their research results. The local publishing channel refers in particular to the publications and dissertation series of universities and research institutes.

4.2.5 The scientific level is questionable

The panels do not have to accept channels formally satisfying the level 1 criteria if the scientific quality or relevance is questionable from the point of view of Finnish research. For this reason, e.g. questionable (“predatory”) open access journals or other channels that publish scientific papers for payment without proper quality evaluation can be placed on level 0.

4.2.6 Characteristics of predatory journals

The publishing model based on author fees has increased the number of predatory journals and book publishers. Characteristic features of the predator include low-quality webpages, fast processing time of manuscripts, vague topic, aggressive email marketing and lack of contact information. Author fees alone do not make the journal suspicious, but for all kinds of fraud attested in the amount of fees, in the composition of the editorial board, database indexing and impact factor information or the peer review implementation is a good reason to leave the journal in level 0.

4.3 Level 2

Level 2 is the leading category of the Publication Forum classification, which contains only a very limited group of peer-reviewed publication channels. Level 2 can be awarded mainly to the leading international publication channels, but in the social sciences and humanities also leading national language channels.

4.3.1 International channels

Level 2 can be awarded to leading scientific publication channels of the various disciplines that meet the following criteria:

• has a wide reach and respect among the international experts in the field

• researchers from different countries seek to publish their best results

• editors, authors and readers represent various nationalities.

All publication channels meeting these criteria cannot be rated as level 2 but the Panels must choose, within their own level 2 quota (see below), the publication channels attracting and selecting the highest quality publications as a consequence of extensive competition and demanding peer reviews. Finnish channels aimed at international audiences are evaluated in relation to other international publication channels.

4.3.2 Domestic Finnish and Swedish language channels

In humanities and social sciences (Panels 14, 16-23), level 2 can also include leading Finnish or Swedish-language publication channels having the widest coverage of the research on Finnish society, culture or history in their field. Due to the lack of citation data, the scientific impact of the domestic publication channels cannot be measured but the level 2 publication channels need to meet the following criteria:

• The quality assessment of the scientific writings must be in line with the best practices;

• The publication series must be among the ones that cover the research in the respective discipline – and the book publishers in their main discipline – most comprehensively and be used by the entire national scientific community in that particular discipline;

• The context of the research problems is strongly focused on the Finnish society or the Finnish or Swedish-speaking culture.

• Publishing in these channels is regarded as high merit as publishing in foreign level 2 channels.

All publication channels meeting these criteria are no rated as level 2 but it will only include a selection of the highest quality and most comprehensive Finnish and Swedish-language publication channels covering the disciplines where it is justified to produce and publish new scientific information in the national languages. The Panel Chairs acting in corpore are to make a consensus decision on the classification of Finnish and Swedish-language publication channels in level 2.

4.4 Level 3

Level 3 is the top category of the Publication Forum classification. Level 3 is a sub-category of the level 2, so level 3 channels need also to meet the level 2 criteria. Level 3 can include the topmost publication series of various disciplines meeting also the following criteria:

• The research published in them represents the highest level in the discipline and has very high impact (e.g., as measured through citation indicators);

• The series cover the discipline comprehensively, not limiting to the discussion of narrow special themes;

• Both the authors and readers are international and the editorial boards are constituted by the leading researcher in the field;

• Publication in these journals and series is highly appreciated among the international research community of the field.

The Panels will prepare a proposal of publication series to be rated as level 3, and the Panel Chairs take consensus decisions to include publication series in the level 3 category.

4.5 Quotas of level 2 and 3

The quotas of levels 2 and 3 are determined for journals and series in each Panel on the basis of the size of the publication channels, in other words, their annual publication volume (see 4.5.1). The aggregate publication volume of the journal/series titles rated as level 2 must not exceed 20% of the corresponding publication volume of the journal/series titles in the panel’s list meeting level 1 criteria. The aggregate publication volume of the journal/series titles rated as level 3 must not exceed 25% of the corresponding publication volume of the journal/series titles in the panel’s list meeting level 2 criteria. In other words, the level 2 consists of 15 % and the level 3 of 5 % of the publication volume of the panel. The book publisher list is common to all panels, and to level 2 a total of about 100 publishers can be classified and to a total of about 10 publishers can be classified as level 3. The panels make suggestions for level 2 and 3 book publishers, and the chairs of all panels confirm the classification by common decision.

4.5.1 Counting the publication volume

The Secretariat will provide the Panels with the publication volume data while the evaluation website will take care of the statistics within the Panel per each discipline. Publication volumes (3-year average) are updated to all Scopus and Web of Science indexed publication series. For other publications volume determined in 2014 will be used, or if the channel has been added to level 1 after 2014, the median of the level 1 journals/series of the panel is used. The publication series consumes the level 2 or 3 quota only up to 2500 publication volume, even if its publishing volume exceeds 2500. The publication volumes are rounded up to the nearest twenty-five. Also the removal of conferences from the panels’ lists of journals/series in 2016 is compensated.

4.6 Considerations to be taken into account in the evaluation

The panels need to have a discussion about the importance of impact factors, as well as the Norwegian and Danish ratings, in the evaluation of publication channels in their field. There also needs to be a discussion about how to deal with review journals in the panel and how open access is taken into account in the assessment.

4.6.1 International impact and prestige

Depending on the discipline, the impact and prestige of the publication channels in the international scientific community can be estimated using the impact factors and level rating indicators that the secretariat provides for the use of panels. The citation databases cover better the natural and medical sciences than the journals/series of engineering, social sciences and the humanities. In case of SSH, the focus of citation databases in English newspaper articles is also problematic. The Norwegian and Danish panel ratings create a more comprehensive basis for evaluating international prestige of journals and series. None of the indicators, however, covers all publishing channels to be evaluated in the Publish Forum.

4.6.1.1 Impact factors

Various impact factors are available for the journals indexed in the Web of Science or Scopus databases, most importantly Journal Impact Factor (JIF), CiteScore, Source Normalized Impact per Paper (SNIP), and SCImago Journal Rank (SJR). Common to these indicators is that they try to measure the impact of publication channels based on the number of references they have received in their publications. The basic assumption is that the larger the number of citations to publications in a journal is, the greater its impact on the international scientific community. In the use of the impact factors, it should be noted that they are not comparable between different disciplines.

4.6.1.2 Norwegian and Danish level ratings

In addition, the panels have the ratings given for journals, series and book publishers in the Norwegian and Danish panel evaluation that is comparable to the Publication Forum. Common to these classifications is that they are also based on an expert assessment, although impact factors and may be used to inform the evaluation depending on the discipline. Norway and Denmark have a 2-tier rating, where class 1 is basic and class 2 is the leading level. In Denmark, there are about 3000 journals/series in Level 2, in Norway about 2000.

4.6.2 Perceptions and suggestions of the research community

Members of the academic community have the opportunity to give panels feedback on ratings by proposing changes to the ratings on the Publications Forum website and on the JUFO portal. In addition, members of the Finnish research community can each determine the 10 most important journals/series and the book publisher for their own research on the JUFO portal. There is also expertise concerning the evaluation of channels belonging to each panel in other panels, and panelists can also make rating suggestions across panel boundaries. Panels can use all of this information to support their assessment. Panel members represent the national research community of their own discipline or research area, and the assignment also includes the need to consult with their background communities.

4.6.3 Review journals

In natural and medical sciences review articles typically gain the most citations because of their nature, so the impact factors for journals that only publish reviews are higher than those publishing original research papers. The best review journals can be classified as levels 2 and 3, as long as they do not fill the entire quota.

4.6.4 Open access

Open access journals are accepted to the rating and are evaluated on the same criteria as traditional publication series. For the levels 2 and 3, the aim is to promote the open science in the following way: Green OA channel can be favored if the embargo is at most 6 months in natural sciences and engineering, and 12 months in SSH. Also open data can be taken into account.”

• if candidates to the level 2 or 3 in the same field have equal impact or prestige, open access journal or the one permitting self-archiving of the peer-reviewed version of the manuscript is chosen to the higher level over the channel that does not support open access.

• the channel allowing for self-archiving can be favoured in comparison if the embargo period is in science and technology for six months and in human sciences for 12 months.

• openness of data can also be considered an advantage. It is checked in the case of comparison whether open access provides grounds to decide selection.

4.6.4.1 Open access publication channels (Gold open access)

Immediately open (gold open access) channels are journals, whose all content is open to all readers. Typically, their operating costs are covered by article processing charges (APCs). APC is not a reason enough to leave open access channel to category 0. Quite a trustworthy register of quality open access journals is a Directory of Open Access Journals, which is integrated into the Publish Forum database. However, it should be noted that not all journals listed in DOAJ meet the criteria for Level 1, for example, regarding the composition of the editorial board and authorship.

4.6.4.2 Channels permitting self-archiving (Green open access)

Channels permitting self-archiving (green open access) are traditional subscription-paid journals that allow articles to be stored, for example, in the University's open publication archive. The SHERPA/RoMEO service, which is integrated into the Publish Forum database, can be used to find a self-archiving policy. In general, the color code alone does not adequately describe the policy, but the journal specific information must be checked from SHERPA / RoMEO.

4.6.4.3 Hybrids

A hybrid open access journal is a subscription journal in which some of the articles are open access. This status typically requires the payment of a publication fee (APC) to the publisher. Hybrid journals allowing self-archiving can be favored according to the present policy explained in chapter 4.6.4., if the embargo is reasonable. Panels can evaluate whether the open access policy of the hybrid journal is sufficient in individual cases.

4.6.5 Representativeness of different fields on levels 2 and 3

The panels are wide-ranging, so they have to be responsible for fair representation of channels from different fields in level 2 and 3. The panels need to look at the journals/series and conferences of each of the research disciplines as a whole in the disciplines of the Web of Science, Scopus, Norway, Denmark and ERIH Plus, or other groupings. Considerations should also be given to those journals belonging to the same category of these disciplines that are placed in the lists of other panels. The level 2 and 3 shares of publication volumes are monitored within disciplines, so that special attention can be paid to areas with significant under- or over-representation on the levels 2 and .

4.6.6 Closed society channels

If the lack of the membership of a certain society considerably restricts the publication opportunity of the researchers, the panels can take it into account while making the ratings on levels 2 and 3. This concerns so called Closed society publication channels, such as CIRP annals and journals of ASME. If this factor has an effect of the rating, the grounds need to be entered in the portal.

5 Transparency and ethical aspects of evaluation

All channels regarded as leading by researchers do not have a level 2 or 3 rating, or the rating of the journals does not fully correspond to the impact factor ranking order of a particular discipline. Often, this is due to the structure of the rating, for example, the level quotas or the classification criteria. However, the evaluation work of the panel of experts in the publication forum should be justified and consistent, and take into account ethical considerations, thereby increasing the transparency and reliability of the rating.

5.1.Transparency of the evaluation

The panels are required to produce more precise justification of what level 1 criteria the channels rated at level 0 do not meet (see 3.2). For Levels 2 and 3, panels need to produce on a general level the principles and criteria on basis of which the classification is made. In addition, the aim is to record channel-specific justifications for decisions that significantly deviate from the impact factors or the Norway and Denmark's ratings of expert panels:

• If the channel gets level 2 or 3, even if the channel is classified in Norway and Denmark for level 1, or the channel gets level 1 even though it is classified in level 2 in Norway and Denmark.

• If the channel receives a higher level than another channel representing the same field of research, the impact of which in the international scientific community is significantly higher on the basis of the indicators.

5.2 Ethical aspects

The members of the panels are representatives of the national research community in their field of expertise. This means that panelists are not only guardians of interests of their own university / unit or a particular publishing channel (publisher or journal).

5.2.1 Responsible conduct of research

The evaluation work must be based on good scientific practice (honesty, general care and accuracy), including the announcement of engagements. Panel members are responsible for complying with the National Board of Research Ethics Guidelines for Responsible conduct of research, the principles of which are universally accepted by universities and research institutes.

5.2.2 Announcement of engagements

Panelists must announce their engagements each time a panel discusses evaluation of channels linked to them. This applies to publishing channels where the panelists have published more than once in the past 5 years or have been journalists or members of the Editorial Board. In these cases, the panelist will not be excluded from the decision-making, but the panel can draw attention to the engagements in the assessment situation. The Secretariat will compile information on the engagements.

6 Panels’ work

The most important task of panels is the classification of publications channels in their fields. Evaluation work is done on the JUFO portal or panel meetings. During its four-year term, the panels will also carry out a yearly complementary evaluation in which previously unclassified or level 0 rated channels can be added to the level 1, as well as a review of ratings, in which the levels 2 and 3 are updated.

6.1 Complementary evaluation

A continually ongoing complementary evaluation, in which panel has 2 months’ time-window to make a decision to place a channel on the level 0 or 1, is implemented entirely through the JUFO portal without panel meetings. The complementary evaluation includes publishing channels that have been proposed to be added to the classification through the Publications Forum web pages or identified from the publication data reported by the research organizations to the ministry of education and culture.

6.2 Review of ratings

The progress of the review of ratings requires panel meetings, in which the evaluation criteria and changes to the level 2 and 3 classification can be discussed. The reviewed rating is validated through the JUFO portal.

6.3 JUFO-portal

In the JUFO portal, panelists can view information about the publishing channels that are being evaluated by the panel, make proposals about the level of publication channels, and make notes about publishing channels. Panelists can also download their channel list as an Excel file.

6.4 Panel meetings

The panels meet at the House of Science and Letters three times during the first year of their term of office, when the task is to carry out a review of ratings. For the last three years of its term of office, when panels perform exclusively a complementary evaluation, the panels meet once a year (September-October).

6.5 Corrections to levels 2 and 3 between reviews

In those years when an review or ratings is not carried out, the panels meet once a year to address feedback and suggested corrections to the ratings. Individual changes to Levels 2 and 3 can be made according to the current volume situation. That is, if a free level 2 and 3 quota has been created dut to addition of level 1 journals, it may be used for upgrades.

6.6 Changes to the panel composition

If there is a need to change or complement the panel composition during the term of office, the Secretariat and the Chair of the Steering Group will prepare a presentation to the Steering Group on possible panelists. Depending on the timetable, the Steering Group appoints panel extension members either at their meeting or at a separate e-mail meeting.

7 APPENDIX 1: Evaluation timeline for the term 2018-2021