1 (9)
PUBLICATION FORUM MANUAL FOR PANELS
2 General
2.1 Publication Forum
The Publication Forum established in 2010 under the auspices of the Federation of Finnish
Learned Societies (TSV) is a system to support the quality assessment of scientific publication
activities in all disciplines, based on the classification of domestic and foreign publication series
and book publishers. The rating decisions are taken by the over 200 expert members of the 23
Evaluation Panels covering various disciplines, universities and research institutes. The Panel
members are nominated on basis of proposals made by the universities, government research
institutes, learned societies and science academies.
3 (9)
2.2 Steering-group
The Panels are appointed and the evaluation work supervised by the Publication Forum Steering
Group, nominated by the TSV Board and constituted by representatives of the Ministry of
Education and Culture (MinEdu), the Finnish Council of University Rectors (UNIFI), the Academy of
Finland, the Council for Finnish University Libraries (SYN), the National Library of Finland, the IT
Centre for Science (CSC) as well as various major disciplines. The work of the Steering Group and
the Evaluation Panels is prepared and supported by the Publication Forum Secretariat employed
by TSV which also provides the Panels with meeting facilities at the House of Science and Letters in
Helsinki. The IT services required by the Publication Forum are run in collaboration with CSC.
2.3 Levels
In the publication forum, the scientific publishing channels are divided into three categories: basic
scientific channels (level 1), leading scientific channels (level 2) and top scientific channels (level 3).
In addition, all publishing channels evaluated by panels that do not meet the minimum criteria for
Level 1 are listed in Category 0. Category 0 does not give the publication channel any special
status, but only indicates that the channel did not meet the criteria for Level 1 at the time of
assessment.
2.4 Publication Forum functions
The publication forum rating serves as an indicator of the quality of university publication
production in the ministry of education and culture’s funding model from 2015 onwards.
Combined with publication data collected by MinEdu, the classification provides a comprehensive
and comparable information on the volume and development of the publishing activities of
organizations and disciplines. The classification disseminates information on the quality, impact
and prestige of scientific journals, series and book publishers in the scientific community, and thus
encourages Finnish researchers to publish their most important research in the leading national
and international scientific publications channels in their field.
2.5 Use of Publication Forum classification in evaluations
In the internal use of research organisations, the classification is best suited for the follow-up of
the progress made within the research area’s or unit’s own publication operations. The results can
be used as information to the expert panels evaluating their research. Using the classification
mechanistically to allocate financing to research units may lead to inequitable treatment of
different fields of science. Due to the differences in publication practices and research questions
and methods, the number of publications on the higher level publication channels produced by
individual researchers varies both between and inside the disciplines and fields of research. The
publication channels of Level 1 may contain individual publications that are above the average in
terms of quality and impact, while the publication channels classified as Level 2 and 3 may include
publications that do not meet the average quality and may be never cited. For these reasons, the
classification cannot substitute for peer evaluation as a criterion or grounds for decisions on an
individual researcher's recruitment, financing or rewarding.
4 (9)
3 Scientific publications and publication channels
3.1 Scientific publication
‘Publication’ refers to a written presentation of the research outcome by an individual researcher
or group of researchers, published as an article in a journal, congress proceedings, as a part of a
compilation or an independent monograph, depending on the discipline. To meet the criteria of
‘scientific’, the publication must present new research outcome in a form to be accessible,
repeatable and usable by the scientific community. The main target group of a scientific
publication is constituted by the experts in the discipline. In the publication type classification
used by the ministry of education and culture, scientific publications correspond to types A and C.
3.2 Scientific publication channel
A ‘scientific publication channel’ refers to printed and digital publication series as well as to book
publishers specialised in the publication of scientific research outcomes: they have an editorial
board constituted by experts, and the publication calls for quality evaluation recognised by the
scientific community, above all referee/peer evaluation. Scientific publication channels are
evaluated primarily in terms of their scientific impact, not their societal impact. Scientific
publication channels may also produce not-peer-reviewed publications, such as comments,
reviews and handbooks. In Publication Forum publication channels are divided in three types that
are journals/series, conferences and book publishers.
3.2.1 Journals/series
A publication series can be a scientific journal, book series or a series of proceedings of a regularly
organised conference. Journals, book series and proceedings with ISSN are classified using the title
recorded in the international ISSN centre. The main rule is that the book series of Finnish book
publishers are classified separately in the Publication Forum, while the book series of foreign book
publishers are not classified.
3.2.2 Conferences
In addition, the classification includes a limited number of conferences listed under the name of
the event, the type of channel in this case being "conference". The panel 2 (Computer and
information science) and 9 (Electrical and Electronic engineering, information engineering) can
classify a conference separately in cases where the level of the main publishing channel, ie the
publisher (e.g. IEEE or ACM) or the publication series (e.g. LNCS), does not reflect the true level of
the publications of the specific conference.
Conferences that have their own publication series with ISSN that only publish proceedings of this
conference will be treated as journals/series with the name registered at the ISSN Center (only if
the ISSN is actually included in the publications). For conferences that do not have their own
(active) ISSN-recognized publishing series, either a publication series with ISSN (in the case of a
journal or a book series specializing in proceedings) or an ISBN-based book publisher is primarily
identified as a publishing channel.
5 (9)
3.2.3 Book publishers
Book publishers are listed under the name in the ISBN registry or under other established name. In
the Publication Forum and the data collection the “book publisher” means a publisher who is
responsible for the publisher's ISBN (not other kind of publisher or printing house). Publishers who
use self-publishing ISBNs are not primarily treated as book publishers. The imprint of the parent
company is classified as an independent publication channel. Publisher’s activities in different
countries are considered to belong to the same publication channel. If the faculties or institutes
have their own ISBNs, they are considered the same publication channel as the university.
3.3 Special cases
3.3.1 New publication channels
As a rule, publication channels will not be evaluated by the panels if publication activities are less
than one year old, or if there are not enough issues or publications in the channel that allow
content to be evaluated.
3.3.2 Channels without ISSN or ISBN
Publication channel must have a registered ISSN or ISBN number. The exception is a small number
of conferences categorized under an established name that do not have their own ISSN-recognized
publishing series.
3.3.3 Professional and popular channels
Publication Forum does not classify channels targeted at professional or general audiences. If the
boundary is not clear, channels for the professional and the general public can be evaluated by the
panels. If the panels find that such channels are not scientific publication series or book publishers,
they will be transferred to a separate list of professional and general publication channels.
3.3.4 Descriptors
Nature and Elsevier have started publishing channels to support open access and reuse of research
data. In the articles, the researchers describe the published data, including the methods and
technical analyzes used for generating/collecting the data. Journals publishing only data
descriptions are placed directly in level 0 without evaluation, but the development of the status of
these channels is monitored.
3.3.5 Software publishing channels
In computer science publishing channels have been set up for publishing open source software
and descriptive articles. Journals specializing in articles describing software are normally processed
for panel evaluation.
3.3.6 Short Monographs
International publishers, for example, Palgrave Macillan and Routledge, have launched a new
peer-reviewed publication format, the publication type of which corresponds to a monograph but
the length of manuscripts is between an article and a traditional monograph. In the MinEdu data
collection short monographs are instructed to be classified as book articles. If the publication
6 (9)
channel for short monographs can be identified with an ISSN, it can be evaluated in the Publish
Forum.
3.3.7 Distribution of publication channels between panels
The journals/series and conferences are divided between the panels according to the discipline, so
that the responsibility for evaluating each publication channel is limited to one panel. The
exception to this rule is a small number of multidisciplinary journals (e.g. Nature, Science and
PNAS). These multidisciplinary publications series, as well as most of the book publishers, have
been placed on a panel of general publication channels (panel 24), which is common to all panels.
All panels can participate in evaluation of the channels placed in Panel 24.
4 Classification criteria applicable to publication channels
4.1 Publication Forum coverage
The Publication Forum includes active publication channels that specialize in publishing of
research results and are relevant from the point of view of Finnish research. The classification thus
covers the leading and internationally renowned journals/series and book publishers of the
various disciplines as widely as possible, but national and local publication channels in different
countries can be limited to those used by or relevant to Finnish researchers.
4.2 Level 1
Level 1 is the basic category of the Publication Forum classification that contains vast majority of
foreign and domestic peer-reviewed publication channels. Open access journals are accepted to
the classification and evaluated using the same criteria applicable to conventional publication
series. Level 1 can be awarded to domestic and foreign journals/series, conferences and book
publishers meeting the criteria of a scientific publication channel:
• The channel is specialised in the publication of scientific research outcomes;
• There is an editorial board constituted by experts;
• The scientific publications are subject to a peer evaluation focusing on the entire
manuscript.
In principle, a publication channel meeting the criteria of a scientific publication channel must not
be included in level 1 if
• over half (1/2) of the reviewers or authors come from the same research organisation that
is also the publisher;
• the scientific level or relevance raises questions.
4.2.1 Specialization in the publication of scientific research results
To level 1 can be accepted channels that publish peer-reviewed publications on a regular basis.
Channels that occasionally publish individual scientific publications will not be accepted for level 1.
Scientific publishing channels may also include not-peer-reviewed publications for both the
professionals and the general public. However, it is recommended that the classification channels
7 (9)
that are acceptable to the level 1 indicate as clearly as possible which publications are peerreviewed.
4.2.2 An editorial team of science specialists
Level 1 channels must have an editorial board including specialists in the field working in
universities or research institutes.
4.2.3 Peer Review
Peer Review refers to a procedure whereby a journal, conference or book publisher invites experts
in the field to make an evaluation of the scientific merits and fitness for publication of manuscripts
offered for publication. The peer review should focus on the entire manuscript to be published,
not just of the abstract.
4.2.4 Local channels
Only such channels can be approved for Level 1, in which researchers from different research
organizations publish their research results. The local publishing channel refers in particular to the
publications and dissertation series of universities and research institutes.
4.2.5 The scientific level is questionable
The panels do not have to accept channels formally satisfying the level 1 criteria if the scientific
quality or relevance is questionable from the point of view of Finnish research. For this reason, e.g.
questionable (“predatory”) open access journals or other channels that publish scientific papers
for payment without proper quality evaluation can be placed on level 0.
4.2.6 Characteristics of predatory journals
The publishing model based on author fees has increased the number of predatory journals and
book publishers. Characteristic features of the predator include low-quality webpages, fast
processing time of manuscripts, vague topic, aggressive email marketing and lack of contact
information. Author fees alone do not make the journal suspicious, but for all kinds of fraud
attested in the amount of fees, in the composition of the editorial board, database indexing and
impact factor information or the peer review implementation is a good reason to leave the journal
in level 0.
4.3 Level 2
Level 2 is the leading category of the Publication Forum classification, which contains only a very
limited group of peer-reviewed publication channels. Level 2 can be awarded mainly to the leading
international publication channels, but in the social sciences and humanities also leading national
language channels.
4.3.1 International channels
Level 2 can be awarded to leading scientific publication channels of the various disciplines that
meet the following criteria:
• has a wide reach and respect among the international experts in the field
• researchers from different countries seek to publish their best results
8 (9)
• editors, authors and readers represent various nationalities.
All publication channels meeting these criteria cannot be rated as level 2 but the Panels must
choose, within their own level 2 quota (see below), the publication channels attracting and
selecting the highest quality publications as a consequence of extensive competition and
demanding peer reviews. Finnish channels aimed at international audiences are evaluated in
relation to other international publication channels.
4.3.2 Domestic Finnish and Swedish language channels
In humanities and social sciences (Panels 14, 16-23), level 2 can also include leading Finnish or
Swedish-language publication channels having the widest coverage of the research on Finnish
society, culture or history in their field.
Due to the lack of citation data, the scientific impact of the domestic publication channels cannot
be measured but the level 2 publication channels need to meet the following criteria:
• The quality assessment of the scientific writings must be in line with the best practices;
• The publication series must be among the ones that cover the research in the respective
discipline – and the book publishers in their main discipline – most comprehensively and be
used by the entire national scientific community in that particular discipline;
• The context of the research problems is strongly focused on the Finnish society or the
Finnish or Swedish-speaking culture.
• Publishing in these channels is regarded as high merit as publishing in foreign level 2
channels.
All publication channels meeting these criteria are no rated as level 2 but it will only include a
selection of the highest quality and most comprehensive Finnish and Swedish-language
publication channels covering the disciplines where it is justified to produce and publish new
scientific information in the national languages. The Panel Chairs acting in corpore are to make a
consensus decision on the classification of Finnish and Swedish-language publication channels in
level 2.
4.4 Level 3
Level 3 is the top category of the Publication Forum classification. Level 3 is a sub-category of
the level 2, so level 3 channels need also to meet the level 2 criteria. Level 3 can include the
topmost publication series of various disciplines meeting also the following criteria:
• The research published in them represents the highest level in the discipline and has very
high impact (e.g., as measured through citation indicators);
• The series cover the discipline comprehensively, not limiting to the discussion of narrow
special themes;
• Both the authors and readers are international and the editorial boards are constituted by
the leading researcher in the field;
9 (9)
• Publication in these journals and series is highly appreciated among the international
research community of the field.
The Panels will prepare a proposal of publication series to be rated as level 3, and the Panel Chairs
take consensus decisions to include publication series in the level 3 category.
4.5 Quotas of level 2 and 3
The quotas of levels 2 and 3 are determined for journals and series in each Panel on the basis of
the size of the publication channels, in other words, their annual publication volume (see 4.5.1).
The aggregate publication volume of the journal/series titles rated as level 2 must not exceed 20%
of the corresponding publication volume of the journal/series titles in the panel’s list meeting level
1 criteria. The aggregate publication volume of the journal/series titles rated as level 3 must not
exceed 25% of the corresponding publication volume of the journal/series titles in the panel’s list
meeting level 2 criteria. In other words, the level 2 consists of 15 % and the level 3 of 5 % of the
publication volume of the panel.
The book publisher list is common to all panels, and to level 2 a total of about 100 publishers can
be classified and to a total of about 10 publishers can be classified as level 3. The panels make
suggestions for level 2 and 3 book publishers, and the chairs of all panels confirm the classification
by common decision.
4.5.1 Counting the publication volume
The Secretariat will provide the Panels with the publication volume data while the evaluation
website will take care of the statistics within the Panel per each discipline.
Publication volumes (3-year average) are updated to all Scopus and Web of Science indexed
publication series. For other publications volume determined in 2014 will be used, or if the
channel has been added to level 1 after 2014, the median of the level 1 journals/series of the
panel is used. The publication series consumes the level 2 or 3 quota only up to 2500 publication
volume, even if its publishing volume exceeds 2500. The publication volumes are rounded up to
the nearest twenty-five. Also the removal of conferences from the panels’ lists of journals/series in
2016 is compensated.
4.6 Considerations to be taken into account in the evaluation
The panels need to have a discussion about the importance of impact factors, as well as the
Norwegian and Danish ratings, in the evaluation of publication channels in their field. There also
needs to be a discussion about how to deal with review journals in the panel and how open access
is taken into account in the assessment.
4.6.1 International impact and prestige
Depending on the discipline, the impact and prestige of the publication channels in the
international scientific community can be estimated using the impact factors and level rating
indicators that the secretariat provides for the use of panels. The citation databases cover better
10 (9)
the natural and medical sciences than the journals/series of engineering, social sciences and the
humanities. In case of SSH, the focus of citation databases in English newspaper articles is also
problematic. The Norwegian and Danish panel ratings create a more comprehensive basis for
evaluating international prestige of journals and series. None of the indicators, however, covers all
publishing channels to be evaluated in the Publish Forum.
4.6.1.1 Impact factors
Various impact factors are available for the journals indexed in the Web of Science or Scopus
databases, most importantly Journal Impact Factor (JIF), CiteScore, Source Normalized Impact per
Paper (SNIP), and SCImago Journal Rank (SJR). Common to these indicators is that they try to
measure the impact of publication channels based on the number of references they have
received in their publications. The basic assumption is that the larger the number of citations to
publications in a journal is, the greater its impact on the international scientific community. In the
use of the impact factors, it should be noted that they are not comparable between different
disciplines.
4.6.1.2 Norwegian and Danish level ratings
In addition, the panels have the ratings given for journals, series and book publishers in the
Norwegian and Danish panel evaluation that is comparable to the Publication Forum. Common to
these classifications is that they are also based on an expert assessment, although impact factors
and may be used to inform the evaluation depending on the discipline. Norway and Denmark have
a 2-tier rating, where class 1 is basic and class 2 is the leading level. In Denmark, there are about
3000 journals/series in Level 2, in Norway about 2000.
4.6.1.3 Mechanical scoring
To be described
4.6.2 Perceptions and suggestions of the research community
Members of the academic community have the opportunity to give panels feedback on ratings by
proposing changes to the ratings on the Publications Forum website and on the JUFO portal. In
addition, members of the Finnish research community can each determine the 10 most important
journals/series and the book publisher for their own research on the JUFO portal. There is also
expertise concerning the evaluation of channels belonging to each panel in other panels, and
panelists can also make rating suggestions across panel boundaries. Panels can use all of this
information to support their assessment. Panel members represent the national research
community of their own discipline or research area, and the assignment also includes the need to
consult with their background communities.
4.6.3 Review journals
In natural and medical sciences review articles typically gain the most citations because of their
nature, so the impact factors for journals that only publish reviews are higher than those
publishing original research papers. The best review journals can be classified as levels 2 and 3, as
long as they do not fill the entire quota.
11 (9)
4.6.4 Open access
Open access journals are accepted to the rating and are evaluated on the same criteria as
traditional publication series. For the levels 2 and 3, the aim is to promote the open science in the
following way: Green OA channel can be favored if the embargo is at most 6 months in natural
sciences and engineering, and 12 months in SSH. Also open data can be taken into account.”
• if candidates to the level 2 or 3 in the same field have equal impact or prestige, open
access journal or the one permitting self-archiving of the peer-reviewed version of the
manuscript is chosen to the higher level over the channel that does not support open
access.
• the channel allowing for self-archiving can be favoured in comparison if the embargo
period is in science and technology for six months and in human sciences for 12 months.
• openness of data can also be considered an advantage.
It is checked in the case of comparison whether open access provides grounds to decide selection.
4.6.4.1 Open access publication channels (Gold open access)
Immediately open (gold open access) channels are journals, whose all content is open to all
readers. Typically, their operating costs are covered by article processing charges (APCs). APC is
not a reason enough to leave open access channel to category 0. Quite a trustworthy register of
quality open access journals is a Directory of Open Access Journals, which is integrated into the
Publish Forum database. However, it should be noted that not all journals listed in DOAJ meet the
criteria for Level 1, for example, regarding the composition of the editorial board and authorship.
4.6.4.2 Channels permitting self-archiving (Green open access)
Channels permitting self-archiving (green open access) are traditional subscription-paid journals
that allow articles to be stored, for example, in the University's open publication archive. The
SHERPA/RoMEO service, which is integrated into the Publish Forum database, can be used to find
a self-archiving policy. In general, the color code alone does not adequately describe the policy,
but the journal specific information must be checked from SHERPA / RoMEO.
4.6.5 Representativeness of different fields on levels 2 and 3
The panels are wide-ranging, so they have to be responsible for fair representation of channels
from different fields in level 2 and 3. The panels need to look at the journals/series and
conferences of each of the research disciplines as a whole in the disciplines of the Web of Science,
Scopus, Norway, Denmark and ERIH Plus, or other groupings. Considerations should also be given
to those journals belonging to the same category of these disciplines that are placed in the lists of
other panels. The level 2 and 3 shares of publication volumes are monitored within disciplines, so
that special attention can be paid to areas with significant under- or over-representation on the
levels 2 and .
12 (9)
5 Transparency and ethical aspects of evaluation
All channels regarded as leading by researchers do not have a level 2 or 3 rating, or the rating of
the journals does not fully correspond to the impact factor ranking order of a particular discipline.
Often, this is due to the structure of the rating, for example, the level quotas or the classification
criteria. However, the evaluation work of the panel of experts in the publication forum should be
justified and consistent, and take into account ethical considerations, thereby increasing the
transparency and reliability of the rating.
5.1.1 Transparency of the evaluation
The panels are required to produce more precise justification of what level 1 criteria the channels
rated at level 0 do not meet (see 3.2). For Levels 2 and 3, panels need to produce on a general
level the principles and criteria on basis of which the classification is made.
In addition, the aim is to record channel-specific justifications for decisions that significantly
deviate from the impact factors or the Norway and Denmark's ratings of expert panels:
• If the channel gets level 2 or 3, even if the channel is classified in Norway and Denmark for
level 1, or the channel gets level 1 even though it is classified in level 2 in Norway and
Denmark.
• If the channel receives a higher level than another channel representing the same field of
research, the impact of which in the international scientific community is significantly
higher on the basis of the indicators.
5.2 Ethical aspects
The members of the panels are representatives of the national research community in their field
of expertise. This means that panelists are not only guardians of interests of their own university /
unit or a particular publishing channel (publisher or journal).
5.2.1 Responsible conduct of research
The evaluation work must be based on good scientific practice (honesty, general care and
accuracy), including the announcement of engagements. Panel members are responsible for
complying with the National Board of Research Ethics Guidelines for Responsible conduct of
research, the principles of which are universally accepted by universities and research institutes.
5.2.2 Announcement of engagements
Panelists must announce their engagements each time a panel discusses evaluation of channels
linked to them. This applies to publishing channels where the panelists have published more than
once in the past 5 years or have been journalists or members of the Editorial Board. In these cases,
the panelist will not be excluded from the decision-making, but the panel can draw attention to
the engagements in the assessment situation. The Secretariat will compile information on the
engagements.
13 (9)
6 Panels’ work
The most important task of panels is the classification of publications channels in their fields.
Evaluation work is done on the JUFO portal or panel meetings. During its four-year term, the
panels will also carry out a yearly complementary evaluation in which previously unclassified or
level 0 rated channels can be added to the level 1, as well as a review of ratings, in which the levels
2 and 3 are updated.
6.1 Complementary evaluation
A continually ongoing complementary evaluation, in which panel has 2 months’ time-window to
make a decision to place a channel on the level 0 or 1, is implemented entirely through the JUFO
portal without panel meetings. The complementary evaluation includes publishing channels that
have been proposed to be added to the classification through the Publications Forum web pages
or identified from the publication data reported by the research organizations to the ministry of
education and culture.
6.2 Review of ratings
The progress of the review of ratings requires panel meetings, in which the evaluation criteria and
changes to the level 2 and 3 classification can be discussed. The reviewed rating is validated
through the JUFO portal.
6.3 JUFO-portal
In the JUFO portal, panelists can view information about the publishing channels that are being
evaluated by the panel, make proposals about the level of publication channels, and make notes
about publishing channels. Panelists can also download their channel list as an Excel file.
6.4 Panel meetings
The panels meet at the House of Science and Letters three times during the first year of their term
of office, when the task is to carry out a review of ratings. For the last three years of its term of
office, when panels perform exclusively a complementary evaluation, the panels meet once a year
(September-October).
6.5 Corrections to levels 2 and 3 between reviews
In those years when an review or ratings is not carried out, the panels meet once a year to address
feedback and suggested corrections to the ratings. Individual changes to Levels 2 and 3 can be
made according to the current volume situation. That is, if a free level 2 and 3 quota has been
created dut to addition of level 1 journals, it may be used for upgrades.
6.6 Changes to the panel composition
If there is a need to change or complement the panel composition during the term of office, the
Secretariat and the Chair of the Steering Group will prepare a presentation to the Steering Group
on possible panelists. Depending on the timetable, the Steering Group appoints panel extension
members either at their meeting or at a separate e-mail meeting.
14 (9)
7 APPENDIX 1: Evaluation timeline for the term 2018-2021
2018 Review of ratings Complementary
evaluation
January-
February
1st panel meeting
- organization of the panels
- introduction to panel work
- Publication Forum and evaluation criteria
- panel fields and representativeness
- level 2 and 3 quotas
- JUFO-portal
- timetable
February Steering-group meeting
- changes to panel compositions
- Top-10 indicator
- other issues
April-May 2nd panel meeting
- preliminary proposal for the level 2 and 3 ratings
Meeting of SSH panel chairs
- preliminary proposal for level 2 national language channels
May-
August
Feedback on the preliminary proposal from the research community?
August Steering-group meeting
- examination of the preliminary propsal and feedback
- feedback to panels
September-
October
3rd panel meeting
- examination of the feedback
- final proposal for the level 2 and 3 ratings
November Meeting of SSH panel chairs
- final proposal for level 2 national language channels
Meeting of all panel chairs
- final proposal for level 3 journals, and level 2 and 3 for the panel 24
channels (journals/series and book publishers)
December Steering-group meeting
- confirming the classification
2019
September-
October
1st panel meeting
- feedback and individual corrections to levels 2 and 3
- other issues
2020
September-
October
1st panel meeting
- feedback and individual corrections to levels 2 and 3
- other issues
2021
September-
October
1st panel meeting
- feedback and individual corrections to levels 2 and 3
- other issues