How Promotion Committees Should Evaluate Journals

A journal name is a fast shortcut for a busy committee, but it cannot tell you whether a candidate's paper is sound, important, or used by anyone.

Guide · 10 min read · Directive Publications Editorial Team

Why the journal name is the wrong shortcut

A promotion file can list forty papers, and a committee has perhaps an afternoon. So the journal becomes the shortcut: a recognised title or a high impact factor stands in for reading the work. The shortcut feels objective. It is not measuring what the committee was asked to judge.

The Journal Impact Factor describes the recent citation performance of a journal as a whole. Citations to individual articles inside any one journal vary widely, so the journal's figure says little about the particular paper in front of you. Clarivate, which publishes the metric, states that journal-level indicators are designed to support journal evaluation, "not to assess the performance of individual papers or researchers."

The research assessment reform documents say the same thing more bluntly. The San Francisco Declaration on Research Assessment (DORA) opens with a general recommendation not to use journal-based metrics "as a surrogate measure of the quality of individual research articles, to assess an individual scientist's contributions, or in hiring, promotion, or funding decisions." The Agreement on Reforming Research Assessment, which underpins the Coalition for Advancing Research Assessment (CoARA), commits signatories to abandon inappropriate uses of the Journal Impact Factor and h-index, and to base assessment primarily on qualitative evaluation "for which peer review is central, supported by responsible use of quantitative indicators."

None of this makes the journal irrelevant. It means the committee should ask the journal a much narrower question.

Split the journal question in two

Evaluating journals for promotion goes wrong when separate questions get merged into one number. Keep them apart, on the assessment form if you can.

The first question is a threshold. A venue either passes or raises a concern that needs checking; it earns no extra credit for being famous. The second and third questions are where the committee's judgement belongs, and neither can be answered from a journal name.

A practical rule for the meeting: when someone says a paper is "in a top journal," the chair asks what the paper showed and who has used it. If nobody in the room can answer, the committee has not assessed that paper yet.

What to ask candidates to submit

DORA's recommendation to institutions is to be explicit about the criteria for hiring, tenure, and promotion, "clearly highlighting, especially for early-stage investigators, that the scientific content of a paper is much more important than publication metrics or the identity of the journal in which it was published." Put that sentence into the dossier instructions, not only into a policy page nobody opens.

A workable format asks for a short statement on each of three to five selected outputs:

1. The question and the finding, in two or three plain sentences.
2. The candidate's contribution. Where the journal published a contributor roles statement, such as CRediT, point to it.
3. Why it matters, written for a committee member outside the specialty.
4. Evidence of use: specific papers that build on it, inclusion in a clinical guideline or systematic review, policy documents, or reuse of shared data and code.
5. Rigour and openness: trial or review registration numbers, the reporting guideline followed, and links to data, code, or protocol where they exist.

Tell candidates not to annotate their publication lists with impact factors or quartiles, and say plainly that leaving them out will not count against anyone. For clinical faculty the fifth item carries real weight. A registered, completely reported trial in a specialty journal is better evidence of careful practice than an unregistered study in a famous title. The Hong Kong Principles for assessing researchers, published in 2020, make the same case for valuing responsible research practices, transparent reporting regardless of the results, and open science.

Rules for journal lists for tenure

Many departments keep a list of acceptable or preferred journals. A list saves time, but only when it works as a floor for legitimacy rather than a ranking of quality. Check yours against these rules:

1. Its purpose is stated in one sentence: to confirm venues are legitimate, not to score papers
2. It says how journals were added, who maintains it, and when it was last reviewed
3. It has no tiers that convert journal rank into points
4. Candidates can make a written case for an unlisted venue, assessed on the same checks
5. A journal is judged as it stood when the paper was accepted, not as it stands today
6. Titles that fail legitimacy checks are removed on a fixed review cycle

Field differences are the main trap. Citation habits differ between specialties, so a single impact factor cutoff applied across a medical school treats a paper in a low-citing specialty differently from one in a high-citing specialty, for reasons that have nothing to do with the work. The Leiden Manifesto for research metrics includes "account for variation by field in publication and citation practices" among its ten principles, alongside "protect excellence in locally relevant research." A list built only from international metrics can also exclude national-language journals that serve local clinicians. The CoARA commitments explicitly recognise contributions "irrespective of the language in which they are communicated."

Newer open-access journals

A new journal cannot be judged by its impact factor, because it has not had time to earn one. The metric counts citations in one year to a journal's content from the previous two years, and it is produced for journals in the Web of Science Core Collection. A recently launched title has not built that record, and may not yet be covered at all. The absence of a metric is not evidence of low quality, and a policy that requires one penalises anyone who publishes in a young journal.

Apply the legitimacy checks directly instead:

• Editors can be identified and verified independently, on their institutional pages or ORCID records.
• Peer review is described concretely: the model, who takes the decision, and what reviewers are asked to assess.
• Fees are published before submission, not revealed after acceptance.
• DOIs on recent articles resolve to those articles.
• Indexing claims check out in the index itself, including the Directory of Open Access Journals for open-access titles. A young journal may not have been reviewed yet, so treat a missing listing as a reason to look harder, not as a verdict.
• Membership claims, for example COPE or OASPA, appear on those organisations' own member lists.

If the journal passes, assess the paper on its content like any other. Open access is not a quality signal in either direction.

Publications in predatory journals

Some dossiers will include a paper in a journal that fails those checks. A consensus definition published in Nature in 2019 describes predatory journals and publishers as "entities that prioritize self-interest at the expense of scholarship," marked by false or misleading information, deviation from best editorial and publication practices, a lack of transparency, and/or aggressive and indiscriminate solicitation.

Handle these cases in a fixed order rather than by reaction:

1. Verify before labelling. Ask the library or research office to run the checks and write down what failed. Minutes should record findings, such as "no described peer review, DOIs do not resolve," not verdicts about a named journal.
2. Do not count it as peer-reviewed output. Whatever else is decided, it cannot carry the weight of a paper that went through verifiable review.
3. Ask the candidate for a short explanation. When was it submitted, was there any review, was a fee paid, and were concerns about the journal known at the time?
4. Separate a single mistake from a pattern. One early-career paper submitted in good faith points to missing guidance. A run of papers across several such outlets, especially after guidance was available, is a different finding.
5. Read the work if it matters to the case. The research may be sound even when the venue was not; it simply has not been validated by credible review.

Involve the research integrity office only where there are concerns beyond the venue itself, such as problems with the data or the authorship. The better fix is catching these outlets before a dossier exists, which is the subject of how institutions can spot predatory publishing risks.

Preprints, datasets, and software

DORA asks institutions to "consider the value and impact of all research outputs (including datasets and software) in addition to research publications." CoARA's first commitment likewise recognises outputs beyond journal publications. Committees therefore need a way to assess work that has no journal at all.

Preprints. Assess a preprint on its content and record plainly that it has not been through journal peer review. medRxiv itself describes its preprints as "preliminary reports of work that have not been certified by peer review." Check whether a later peer-reviewed version exists, and whether the preprint has attracted public reviews you can read. Do not penalise a candidate for sharing early, and do not treat a preprint as equivalent to a reviewed paper either. If members are unsure how preprints, accepted manuscripts, and published versions relate, the differences are set out in article versions explained.

Datasets and software. Ask for a persistent identifier, a licence, documentation good enough for someone else to reuse the output, and evidence that someone has: data citations, downstream papers, or adoption by other groups. A well-documented dataset that several teams build on can be a larger contribution than another incremental paper. Candidates can make that evidence traceable by citing data, software, and code formally in their own papers.

Running the committee meeting

Good criteria fail when the meeting falls back on journal names under time pressure. A few procedural rules prevent that:

• Assign readers. Two members read each candidate's selected outputs before the meeting and report on content, not venue.
• Brief external referees the same way. Ask letter writers to comment on the work and its influence, not on where it appeared.
• Name the shortcut when it appears. The chair redirects any argument resting on impact factor or journal prestige back to the paper.
• Minute the reasoning. Record why an output was judged strong or weak, so decisions can be explained and appealed.
• Train members before each cycle. The CoARA commitments include training for assessors on the criteria and processes they apply.
• Look back afterwards. Check whether particular fields, clinical roles, or career paths were consistently rated lower, and find out why.

The first change to make

If you change one thing this cycle, change the dossier instructions. Ask for a handful of selected outputs with a plain account of contribution and use, and state in writing that journal metrics are not a criterion. Once candidates stop building their case around journal names, committees find it much easier to stop reading one.

When a committee does need to check how a journal handles manuscripts, the publisher's own description of its process is the place to start. Directive Publications sets out its peer review process publicly, and further guidance for research offices and authors is on the Directive Academy blog.