Many medical journals route research papers to a statistician as well as subject reviewers. Here is what that second reader checks, in the order they check it.
Who does statistical review, and how often
Statistical review is an assessment by someone with methodological training, typically a biostatistician or epidemiologist, alongside the subject-matter reviewers. Its question is narrower than general peer review: can the design, analysis and reporting support the conclusions drawn?
It is not universal. In a survey of leading biomedical journals, run in 2017 and published in 2020, 107 of the 364 journals approached gave eligible responses. Of those, 23% said all their research articles received specialised statistical review, 34% used it for between 10% and 50% of articles, and 34% for 10% or fewer. About a third of journals responded and fewer than three in ten gave usable answers; the authors warn that this low rate raises the prospect of selection bias, "albeit probably towards journals more likely to use statistical review". Read the figures as indicative, and as likely to overstate how common statistical review is.
Two further findings matter for authors. Among journals using statistical review for more than 10% of articles, 72% said the statistical reviewer always or usually sees the revised manuscript, and 73% said it leads to an important change at least half the time.
The practical rule: you usually cannot tell in advance whether your paper will go to a statistician, so prepare as though it will. Even without a dedicated statistical reviewer, the questions below are the ones a careful editor will ask.
Design: the problems reanalysis cannot fix
Design comes first, because a design flaw cannot be repaired by running a different test. Expect these questions:
- Does the analysis answer the stated question? The primary outcome in your methods should match the objective in your introduction and, for a registered trial or published protocol, the primary outcome in that record. If it changed, say when and why.
- Is the sample size justified? Give every input: the target difference and where it came from, the assumed standard deviation or event rate, the significance level, the power, and any allowance for dropout. If a reviewer cannot reproduce your number, expect a query.
- What is the unit of analysis? Eyes within patients, patients within clinics and repeated measurements over time are not independent observations. Analysing 200 eyes from 100 patients as 200 independent data points makes the results look more precise than they are.
- How was confounding handled? In observational studies, explain how adjustment variables were chosen and whether that choice preceded the analysis.
If the study was designed without statistical input and has a weakness here, state it plainly as a limitation. A reviewer will find it either way, and candour reads better than a justification written after the fact.
Methods: enough detail to verify
The benchmark comes from the ICMJE Recommendations: describe statistical methods "with enough detail to enable a knowledgeable reader with access to the original data to judge its appropriateness for the study and to verify the reported results." The same section asks authors to specify software and versions and to distinguish prespecified from exploratory analyses, including subgroups.
The SAMPL guidelines by Lang and Altman, listed by the EQUATOR Network, turn that principle into specifics. The practical point is that a methods section is read as a claim about what you did, and a gap in it is read as a gap in the analysis:
| What is missing | What the reviewer has to assume |
|---|---|
| A named method for each analysis | That they must guess which test produced which number |
| Any statement that assumptions were checked | That they were not, and a parametric test may have been used on skewed or paired data |
| How missing data were handled | A complete-case analysis, with its bias unexamined |
| How outliers were treated, or a variable categorised | That the decision followed seeing its effect on the result |
| Whether multiple comparisons were adjusted for | That significance is optimistic in proportion to the number of tests |
| A prespecified-versus-exploratory label | That analyses were selected after the data were seen |
| Software and version | That the result cannot be reproduced exactly |
None of these assumptions is necessarily correct, which is why they arrive as queries rather than rejections. One clause each closes the question before it is asked.
Also name the reporting guideline for your design. ICMJE points authors to CONSORT for randomised trials, STROBE for observational studies, PRISMA for systematic reviews and STARD for diagnostic accuracy studies; each includes items on statistical reporting.
Results: numbers that can be checked
Much of what comes back at this stage is not a dispute about method: it is a number in one place that does not match the same number somewhere else. Those queries are the cheapest to prevent, because finding them needs no statistical training, only a deliberate pass through your own manuscript:
- Reconcile the denominators. Follow the total enrolled figure through every exclusion, withdrawal and loss to follow-up to the number in each analysis. Every drop needs a reason and a count, and flow diagram and tables must agree.
- Recompute a sample of percentages. Divide numerator by denominator yourself for several per table. Spreadsheet rounding or a revised dataset both leave percentages that no longer match their counts.
- Match the abstract to the tables. The abstract is written last and revised most, so it drifts. Every estimate, interval and count in it should appear identically in the body.
- Check each analysis reports its own N. Analyses with different amounts of missing data have different denominators. State the number contributing to each rather than leaving the reader to assume the headline total.
SAMPL asks for numerators and denominators with every percentage, the sample size for each analysis, and effect estimates given with a measure of precision, usually a 95% confidence interval, at least for primary outcomes. Beyond that, follow the target journal's number style consistently across abstract, text and tables.
One point specific to review: give each table a caption naming the analysis population, the statistic reported and the test used. A reviewer who must reconstruct that from the methods section will ask rather than guess.
Interpretation: conclusions the data must support
Interpretation queries concern the gap between what the analysis produced and what the text claims. Check your discussion and abstract for these:
- Statistical versus clinical significance. ICMJE asks authors to distinguish them, so a conclusion that rests on the word "significant" alone will be queried. Say how large the effect was and why that size matters to patients or practice.
- "No significant difference" presented as "no difference". A non-significant result with a wide confidence interval is compatible with an effect that would matter clinically. Say what the interval includes rather than declaring the question settled. Our post on why null results matter covers how to present such findings without apology.
- Causal language from associations. "Reduced" and "prevented" imply causation an observational design may not support.
- Subgroup findings in the headline. Exploratory subgroup results belong in the results, labelled as such, not in the conclusion.
- Loose statistical words. ICMJE advises against nontechnical uses of "random", "normal", "significant", "correlations" and "sample". SAMPL adds that correlations should not be called low, moderate or high unless you define those ranges.
Code, data and the analysis plan
Requirements here vary by journal and funder. Some require code or data availability; others only encourage it.
For clinical trials, ICMJE's rule is that since 1 July 2018, manuscripts reporting trial results submitted to ICMJE member journals must contain a data sharing statement. Many journals that say they follow the ICMJE Recommendations ask for one too, but not all enforce it, so check your target journal's policy rather than assuming. The statement must say whether deidentified individual participant data will be shared and which data, whether related documents such as the protocol and statistical analysis plan will be available, when and for how long, and on what access criteria. "Undecided" is not an accepted answer on the participant-data question.
Where the journal allows it, sending the statistical analysis plan with the submission is one of the most useful things you can do for a statistical reviewer: it lets them compare what you planned with what you ran, answering the prespecified-versus-exploratory question in one step. Analysis code does the same for details a methods section cannot hold. Under double-blind review, strip author names and institutions from code files and any linked repository. Our guide to citing data, software and code covers giving each a persistent, citable record.
When you disagree with the statistical reviewer
Statistical reviewers can be wrong, or can prefer a method that is reasonable but not the only reasonable choice. Disagreeing is fine; ignoring the comment is not, especially when the same reviewer is likely to see your revision.
The survey authors point out that an editor may not know whether a statistical request is reasonable, or how to settle a dispute between the reviewer and authors with a statistician of their own. So write your response for that editor:
- Restate the concern in one sentence, so it is clear you understood it.
- Explain your reasoning, citing a standard text, reporting guideline or methods paper where one applies.
- Where it is reasonable, run the suggested analysis too and report it as a sensitivity analysis, stating whether your conclusions change.
- Never quietly replace a prespecified primary analysis; label any new analysis as added in response to review.
- Have a statistician uninvolved in the study read your response before you send it.
For the tone and structure of the full response letter, see how to respond to peer review comments.
When each check has to happen
The checks above are not equally urgent, because they are not equally fixable. Sorting them by the last moment at which they can still be addressed is more useful than a flat list:
| Stage | What must be settled here | Cost of leaving it |
|---|---|---|
| Protocol and registration | Primary outcome, unit of analysis, sample size and its inputs, analysis population | Unfixable later: a limitation at best, a rejection at worst |
| Before unblinding or analysis begins | Which analyses are prespecified, how missing data will be handled, which subgroups will be examined | Anything decided after this is exploratory and must be labelled so |
| Writing the manuscript | Methods detail, assumption checks, software and version, guideline checklist | A round of revision, usually straightforward |
| Final read before submission | Denominators reconciled, percentages recomputed, abstract matched to tables | Revision, plus avoidable damage to confidence in the rest |
The unrecoverable failures all sit in the top row, which is the argument for bringing in a statistician at the design stage rather than at the revision. Senior authors should run the final read, being usually the only ones who know which analyses were planned and which were added along the way.
Where to go from here
Statistical review is easiest to pass when the manuscript already answers its questions. Directive Publications describes what its reviewers assess on its peer review process page, and its author guidelines set out what to prepare before you submit.
Frequently asked questions
Do all medical journals use a dedicated statistical reviewer?
No. Practice varies widely between journals. In a survey of leading biomedical journals whose data were collected in 2017 and published in 2020, 23% of responding journals said all their research articles received specialised statistical review, while about a third used it for 10% of articles or fewer. Only a minority of journals answered, so those shares are indicative rather than exact. Prepare your manuscript as though a statistician will read it, because you usually cannot tell in advance.
What statistical problems most often lead to rejection?
There is no single list that applies across journals, but the most serious problems are the ones reanalysis cannot fix. These include a design that cannot answer the stated question, a sample size with no justification, observations treated as independent when they are not, and conclusions the data do not support. Reporting problems, such as missing confidence intervals, are more often handled through revision.
Should authors share their analysis code with reviewers?
Check the journal's policy first, because some require code or data availability and others only encourage it. Sharing code is usually worth doing where it is permitted: it replaces inference with evidence about what was actually run. Make sure the version you share reproduces the numbers in the submitted manuscript, since a mismatch between the two creates a harder problem than not sharing at all.
How should results be presented so a statistical reviewer can check them?
Make every reported number traceable. Give the numerator and denominator behind each percentage, state the sample size contributing to each analysis, and report effect estimates with a measure of precision such as a 95% confidence interval for primary outcomes. Then check that the figures in the abstract match the tables exactly, because mismatches between the two are among the most common queries and the easiest to prevent.
What should I do if I disagree with a statistical reviewer's comments?
Disagree explicitly rather than quietly declining to act, and give your reasoning with a supporting reference where one applies. Keep the language plain enough for an editor without statistical training to follow, because the editor decides and may have no statistician in house to adjudicate. Having a statistician uninvolved in the study read your response before you send it is a good final check.