Training Peer Reviewers at Your Institution

Most researchers learn to review by copying a supervisor or quietly drafting reports for one. A deliberate programme teaches the job in the open, with feedback.

Guide · 11 min read · Directive Publications Editorial Team

Why a lecture is not enough

Most researchers learn to review the way they learned many research skills: by watching someone senior and then doing it. That apprenticeship can work well. It can also mean a postdoc drafts a report that reaches the journal under their supervisor's name alone. A 2019 survey of early career researchers published in eLife found this kind of ghostwriting was common, and most respondents did not consider it ethical.

Formal training is not automatically the fix. A 2023 Cochrane review of 10 randomised trials concluded that training peer reviewers may lead to little or no improvement in the quality of peer review, while noting that most of the evidence was low certainty and that better studies are needed. The interventions it covered ranged from workshops and mentoring to emails, written guidance, videos and checklists.

A more encouraging signal comes from a researcher-led course for doctoral students in Berlin. The evaluation of the Peerspectives course, published in Research Integrity and Peer Review, describes about 18 weeks of teaching: 12 hours of lectures, homework, and 12 hours of small-group workshops in which students reviewed manuscripts under consideration at The BMJ with experienced mentors. BMJ editors, blinded to whether a report was written before or after the course, scored the post-course reports higher. The study had no control group, and the trainees' reports were not shown to match those of the journal's own invited reviewers, so treat it as a design model rather than proof.

The design lesson is consistent: trainees need to write real reports, get specific feedback, and write again. COPE's Ethical Guidelines for Peer Reviewers say they are meant partly as an educational resource for institutions training their students and researchers, which makes them a sensible spine for a programme.

The six steps below build that kind of programme:

1. Set outcomes you can see in a report.
2. Choose who leads it.
3. Build a ladder of formats.
4. Write down the rules for real manuscripts.
5. Score reports and return feedback.
6. Measure what trainees learn.

Set outcomes you can see in a report

"Understand peer review" cannot be assessed. Write each outcome as something a trainee does on the page, then make every session serve at least one of them.

The last three rows reflect obligations in COPE's guidelines. For the reporting row, the EQUATOR Network's toolkit on peer reviewing research makes a point worth teaching early: authors do not have to have done every item in a guideline, but they do have to have discussed it. A trial that could not blind participants is fine if the manuscript says so and explains why.

If a session you are planning does not move one of these outcomes, cut it or move it to a different course.

Choose who leads it

Pair two people. A researcher who reviews regularly in the trainees' field, ideally with editorial experience, supplies the judgement: what counts as a major flaw, which requests are unreasonable, how an editor reads a report. An organiser from the graduate school, research integrity office or library supplies everything else: scheduling, confidentiality agreements, records of who completed what, and course credit.

Plan capacity around mentors, not rooms. Peerspectives set enrolment by the number of workshop mentors available, and its group reports each combined input from four participants and an editor-mentor. Count your mentors first, multiply by a small group size, and cap enrolment there.

Decide where the programme sits. If your doctoral programme already runs a publishing module, as described in our guide to what every PhD programme should teach about scholarly publishing, reviewer training works well as a follow-on for students who have drafted a manuscript of their own. Make it credit-bearing if you can. Peerspectives students could earn credit for their doctoral studies, and an elective with credit is harder to drop in a busy term.

Build a ladder of formats

Start with material that carries no confidentiality risk, and move to live manuscripts only once trainees can write a report an editor could use.

For the orientation, the timed structure in our one-hour publishing ethics workshop adapts well. Assign free self-paced material beforehand so the session goes on cases. Clarivate's Web of Science Academy offers free online courses, including an introduction to peer review, co-reviewing with a mentor, and mentoring in peer review for senior researchers.

For mock reviews, choose papers from journals that publish reviewer reports alongside the article; our guide to transparent peer review explains the models you will find. Have trainees write their report before they read the published ones, then compare: what did the real reviewers raise that the trainee missed, and what did the trainee raise that nobody else did? One caution: the published article has already been revised in response to review. Use the originally submitted version where the journal releases it, and otherwise tell trainees that some problems will already have been fixed.

A partnership with a journal is the top rung, and it needs structure. In Peerspectives, The BMJ's research editors assigned manuscripts to workshop groups, students signed a confidentiality agreement, and the journal routinely tells authors and reviewers about its research programme and lets them opt out. Do not improvise any of that. Agree it in writing with the editor first.

Write down the rules for real manuscripts

Co-reviewing a live manuscript is where most of the learning happens and where the ethical risk sits. COPE's guidelines say reviewers must not involve anyone else in a review, including early career researchers they mentor, without first obtaining permission from the journal. Supervisors must request permission and abide by the editor's decision, and where a student performs the review under guidance, the student should be acknowledged as the reviewer of record. The ICMJE's responsibilities for peer reviewers add that reviewers who get help from a trainee should acknowledge that contribution in their comments to the editor, must request the journal's permission before using AI to help with a review, and should destroy copies of the manuscript after submitting.

Put these rules in the programme handbook as a checklist every mentor signs up to:

1. The invited reviewer asks the editor before the trainee sees anything, and keeps the reply
2. The trainee checks their own conflicts: same institution as an author, recent collaboration, or a similar manuscript in preparation
3. The trainee drafts the full report before the mentor comments
4. The mentor gives reasons for substantive edits, and the trainee makes the revisions
5. Both names reach the editor, in whatever field or comment box the journal provides
6. Nothing is uploaded to an AI tool without the journal's permission
7. Copies are deleted once the review is submitted

A short request makes permission easy to give:

If the editor says no, the trainee does not see the manuscript. Substitute a mock review and move on. If a trainee spots possible misconduct, they tell the mentor, who raises it with the editor in confidence. COPE is explicit that reviewers should not investigate on their own.

Score reports and return feedback

Feedback only works if it is consistent, so use a rubric. The Review Quality Instrument, published in 1999 by van Rooyen, Black and Godlee, scores how far a reviewer comments on five aspects of the manuscript (importance of the research question, originality, strengths and weaknesses of the method, presentation, and interpretation of results) and two aspects of the review itself (constructiveness and substantiation of comments), each on a scale of 1 to 5. Those seven items map closely onto the outcomes in step one, which makes them a practical marking sheet.

Calibrate markers before they mark. Have two markers score the same two reports independently, compare, and agree what a 3 and a 5 look like on each item. It matters: in the Peerspectives evaluation, agreement between pairs of editors using a single overall quality score was only moderate.

Then return feedback in a form the trainee has to act on:

• The scored rubric, with the lowest item flagged
• Two quoted passages from the report: one that worked and one that did not, each with a reason
• A request to revise the report before the next exercise

Keep the revision step. The Peerspectives authors note that earlier studies found short training, written feedback from editors, and simply pairing novices with experienced reviewers to be of limited value in improving review quality. So make trainees do something with the feedback rather than just read it.

Measure what trainees learn

Measure the reports, not the satisfaction forms. A workable design borrows from Peerspectives:

1. Before the programme, each trainee reviews a manuscript alone, to a normal review deadline.
2. After it, each trainee reviews a different manuscript of similar difficulty.
3. Randomise which manuscript each trainee gets at each point, so difficulty evens out.
4. Have markers who did not teach the trainees score both reports without knowing which came first.

Collect self-rated confidence if you like, but record it separately. It tells you how trainees feel, not what an editor would receive.

Know the limits of a before-and-after design. Without a comparison group you cannot separate the course from everything else trainees learned that term. If demand exceeds places, trainees waiting for the next intake can write the same two reports and serve as a comparison group.

Track the longer view too. Keep a log for each trainee of every review they complete, the journal, the date, their role, and the editor's permission email for co-reviews, but never manuscript titles. Over time, reviews completed under a trainee's own name and repeat invitations from editors are the outcomes that matter most.

Start small, then scale

A first run needs one mentor, one organiser, a handful of trainees, two mock reviews, and the permission checklist. Run it, score the reports, and change what did not work before adding a journal partner or a second cohort.

When you build the reading list, pair COPE's guidelines with one real journal's expectations of its reviewers. The Directive Publications peer review page sets out what its reviewers assess and the responsibilities it expects of them, which gives trainees a concrete example to review against.