Negative and Null Results in University Research Culture

A study that finds no effect is not a failed study, and leaving null results unpublished distorts the evidence base and wastes public funds.

Opinion · 12 min read · Directive Publications Editorial Team

The culture problem

Promotion and lab storytelling reward “positive” narratives. Students learn to bury nulls — then the literature becomes lopsided.

Nobody teaches this rule, and everyone learns it. A doctoral student watches which results get a poster, which get a slot at lab meeting, and which sentence a supervisor uses when introducing their work to a visitor. Positive findings get a story. Nulls get "we're still working on that one." Within two years the student has internalised a filter that nobody ever wrote down and that nobody would defend if you said it out loud.

The incentive behind it is real, not imagined. It is not useful to tell early-career researchers that the pressure is in their heads. A null result is harder to place, takes longer to get through review, attracts fewer citations, and is harder to build a grant narrative on. Asking a fourth-year student to absorb all of that personally, for the collective good of the evidence base, is asking them to pay for something the institution benefits from. That is exactly why the fix has to be institutional rather than a matter of individual virtue.

The second effect is on the analysis, not the drawer. When the only publishable outcome is a positive one, pressure lands on the analysis long before it lands on the decision to publish. Nothing that looks like misconduct is required. Drop the outlier that has a defensible reason to be dropped. Add the covariate that plausibly matters. Look at the subgroup where the effect seemed to be there. Report the measure that behaved. Each step is arguable on its own; the sum is a finding that would not survive re-running the study. A researcher who knows that a clean null is publishable has far less reason to walk down that path, which makes crediting null results a misconduct-prevention measure and not only a fairness one.

What it looks like in practice. Ask any established lab how many complete datasets it holds that were never written up — collection finished, analysis finished, no manuscript. The number is rarely small, and the reason is almost never that the work was bad.

Why institutions should care

• Better meta-analysis and replication
• Less wasted duplication
• More honest doctoral training
• Reduced pressure toward questionable analytical flexibility

Better meta-analysis and replication. A meta-analysis can only pool what was published. When no-effect studies stay in drawers, the pooled estimate drifts toward the positive, and the drift is invisible to the reader, because missing studies leave no trace in a reference list. That estimate then feeds a systematic review, the review feeds guidance, and the guidance reaches people who will never read the underlying papers. Replication is distorted in a related way: a team that finds nothing has no way of knowing whether it is the first group to find nothing or the fifth, so it treats its own result as a failure and files it, and the cycle repeats.

Less wasted duplication. An unpublished null does not stay neutral; it invites the next group to spend the same money on the same question. Those costs are usually public, and they are paid in doctoral years as well as in grant lines. Turn it around and the case is stronger still: the money for a completed null study has already been spent, so that information is the cheapest your institution will ever produce. Declining to publish it is discarding something you have already paid for.

More honest doctoral training. A student whose main project returns a null has to be able to write the chapter, defend it, and be employable afterwards. If your department's practical answer is "start something else in year three," you have taught that student that the scholarly record is for wins, and you have quietly extended their degree. The alternative is concrete: allow a null chapter, examine it on design, execution and reporting rather than on the direction of the result, and write that into the thesis guidance so neither the student nor the examiner has to guess.

Reduced pressure toward questionable analytical flexibility. Preregistration, analysis plans and reporting standards all assume a researcher can afford to be held to what they wrote down. If the only acceptable outcome is still a positive one, a prior commitment becomes a form to complete and then depart from, and the departure gets explained away in a sentence nobody checks. The reward is what actually governs behaviour, so the reward is what has to change first.

Academy reading: Directive Academy (including reporting negative/null results) · Open access journals

What universities can do

1. Credit rigorous null results in evaluation
2. Teach preregistration / analysis plans where appropriate
3. Support outlets and article types that welcome null findings
4. Stop treating “no difference” as a career embarrassment

Credit them in the documents, not in the town hall. Saying that the department values null results changes nothing on its own. The change is a sentence in the promotion and probation criteria: where the criteria refer to outputs, state that a well-designed study reporting no effect counts as an output, and is assessed on the quality of its design, execution and reporting rather than on the direction of the finding. Evaluation formats that ask a candidate to nominate a few outputs and explain their contribution make this workable, because a null can be argued for; formats that count publications and citations cannot see it at all. What goes wrong is predictable. The policy is announced, no document is edited, the first candidate to take it seriously is marked down by a panel faithfully applying the written criteria, and everyone in the building learns which of the two statements was real.

Teach the commitment, not the registry. The useful part of preregistration is not the platform. It is deciding the primary outcome, the analysis, the exclusion rules and the sample size before you see the data, and having a dated record that you did. "Where appropriate" is load-bearing here: exploratory, qualitative and discovery work should not be forced into a confirmatory frame, and the honest move in that case is to label the work exploratory and say so in the paper. The training exercise that lands best is to have students write an analysis plan for a study they have already run. They discover for themselves how many decisions they made after seeing the numbers, which is a lesson no lecture delivers.

Name the routes, because "publish it somewhere" is not advice. Registered reports, where a journal reviews the protocol and commits to publication before the results exist, are the strongest structural fix available, because acceptance cannot depend on the direction of the finding. Beyond that: journals that state a null-results policy in their scope, short and brief report formats, data descriptors for a well-documented dataset, replication sections, and a repository or thesis deposit with a persistent identifier so the work is citable and findable even when no journal takes it. Institutional support means budgeting for fees where they apply, accepting these formats on internal forms, and making the repository route easy enough that a departing postdoc can use it in an afternoon.

Change what happens in the room. The last point is cultural, and it is the one that cannot be delegated to a policy. Give the null a lab-meeting slot and a poster. Introduce it as a result. When a supervisor's own study returns nothing and the supervisor writes it up anyway, that does more for the group's behaviour than any document. A workable decision rule: if the question was worth spending the money on, the answer is worth reporting. The time to decide whether a question deserves an answer is before the work, not after you have seen which way it went.

What makes a null result worth publishing

Not every null belongs in the literature, and pretending otherwise weakens the argument. "We found no effect" and "we could not have found an effect" are different claims, and a reader cannot tell them apart unless you show your work.

Report precision, not just absence. Give the effect estimate with its confidence interval and interpret that interval in words. "No significant difference" is a statement about a threshold; the interval tells the reader which effect sizes remain plausible. A narrow interval close to zero is real evidence. A wide interval spanning substantial effects in both directions mostly says the study was too small, and saying so plainly is more useful than a conclusion the data cannot carry.

Show that the measure could have detected something. A positive control, a manipulation check, a known effect reproduced in the same run — some evidence that the assay, instrument or task was working on the day. Without it, a null is indistinguishable from a broken measurement, and a reviewer is right to be unconvinced.

Prespecify wherever the design allows it. A null defined in advance — this outcome, this analysis, this exclusion rule — is far harder to dismiss than one that emerges from a series of choices made after the data arrived.

Frame the question as equivalence when the field cares about a threshold. State, before analysing, the smallest effect that would matter in practice, and ask whether you can rule it out. That turns "we found nothing" into "effects larger than this can be excluded," which is a claim other researchers can use in a design, a review or a guideline.

The decision rule follows from those four. If you can state the smallest effect your design could reliably detect, and it is smaller than the smallest effect anyone would act on, you have a publishable null result. If you cannot, what you have is an inconclusive study — still worth depositing with its data, but written up as inconclusive rather than as evidence of absence.

How to write up a study that found nothing

Say it in the title and the abstract. Hedging the title into "investigating the relationship between X and Y" tells the reader nothing and hides your own contribution. State that no effect was detected, or that the results were consistent with no meaningful difference. Avoid "we failed to find," which describes you rather than the world.

Build the introduction around the question, not the hypothesis. End the introduction with the question and why the answer matters whichever way it goes. If the answer only matters in one direction, the study was never a good use of the grant, and that is worth knowing before you design the next one.

Let the methods carry the weight. This is where a null earns its credibility, so the methods section gets longer than it would be for a positive result, not shorter. Sample size and how it was chosen, blinding, exclusion rules and when they were set, instrument and software versions, and a plain statement of the effect size the design could detect.

Do not rescue the paper with a subgroup. Leading with the one comparison that reached significance while the prespecified primary outcome sits in a supplementary table is the most common way a good null result becomes a bad paper. Report the primary outcome first, label anything exploratory as exploratory, and let it stay exploratory.

In the discussion, separate what you can rule out from what you cannot. "An effect of the size previously reported is inconsistent with these data" is defensible. "There is no effect" usually is not. Reviewers argue less with a paper that states its limits clearly, and an editor can defend it.

For authors

Write null results with the same methods rigor as positive ones. Clear limits beat spun claims.

Same rigor means the same work, in the same order. The temptation with a null is to write it faster and shorter because it feels like a lesser paper. This is the one manuscript where thin methods are fatal, because the whole claim rests on the reader believing you would have seen the effect if it had been there.

If your supervisor is reluctant, make it small and give it a date. Propose a short report rather than a full paper, offer to do the writing yourself, name the target outlet, and set a date for the first draft. Most resistance is not principled objection; it is a reasonable belief that the work will take months and go nowhere. A scoped proposal with a deadline removes that objection.

If the manuscript will take a while, make the work exist in the meantime. Deposit the dataset and the analysis code somewhere that issues a persistent identifier, so the next group searching this question can find it and cite it, and so the record survives your move to another lab.

For Authors · Submit

Where to start this term

If you run a lab, list the completed datasets that were never written up. Sort them by one question: could this design have detected an effect worth caring about? Take the first that clears the bar and write it as a short report. One is enough — the point is to show the group that a null can leave the building.

If you sit on the committee that writes evaluation criteria, do the smaller thing. Find every line that refers to outputs and add the sentence that a rigorous study reporting no effect counts as one. It costs nothing, it takes a single meeting, and it is the change that makes every other recommendation on this page affordable for the people you are asking to make it.

More: Directive Academy · For Authors