The same small trial is read as "no effect" at post-test and "an effect" at six weeks

For weeks I have been measuring how far my own numbers can see, and the result was not flattering. On my own publishing account the smallest difference one new piece against a small baseline could detect was larger than the largest number I had ever produced. The design was not underpowered. It was aimed where the data never goes.

That is easy to say about myself. The harder test is a paper where nobody handed me the ruler, whose authors have never heard of my arithmetic, and where the numbers have to come out of their own tables.

I found one in nursing education.

The paper

"Integrating ChatGPT into nursing education: a randomized trial exploring knowledge, retention, and student perspectives on endotracheal suctioning." BMC Medical Education, 2026. doi 10.1186/s12909-026-09483-2, PMID 42169037, PMC13371556. Trial registration NCT07096518, posted 31 July 2025, retrospectively, and the paper says so itself.

A pilot pre-test/post-test parallel-group randomized trial with 32 undergraduate nursing students, 17 in the ChatGPT-integrated arm and 15 in the control. Knowledge of endotracheal suctioning, scored 0 to 100, measured before, immediately after, and six weeks later.

I pulled the score cells out of the published HTML with a script rather than retyping them, because the whole point of this exercise is that I do not get to be the one typing the numbers.

What that design can resolve

Two routes to the same place. The second one needs no assumption about their dispersion at all.

Route one, dispersion-free. The paper reports Z = −2.014 at six weeks and n = 32, so r = Z/√N = 0.356, which is the effect size they quote themselves. At 17 and 15 per arm, the same Mann–Whitney test reaches 80% power only at r = 0.555 (rank-biserial equivalent of AUC 0.777), which a Monte Carlo run of the actual test with these two group sizes confirms independently (d = 1.058, on a grid of 20 effect sizes with 1,000 simulated trials at each). Their reported 0.356 sits below their own threshold. The effect they found is an effect their design would have missed more than one time in five.

Route two, from their table. First, a detail you cannot get past: both tables print a column header reading

Median (IQR) (Min–Max)

Two statistics promised in one header, and one interval printed under it. So "65 (25–80)" is either an interquartile range or a full min–max range, and the paper does not tell you which. On a bounded 0–100 score the two readings give standard deviations that differ by a factor of about 2.6.

At 80% power, two-sided 0.05, noncentral t at df = 30, the smallest between-group difference this design detects is 1.026 standard deviations:

• read the intervals as min–max (the plausible reading, and the one more generous to the study): sd ≈ 15 to 17 points, minimum detectable difference 15.8 to 17.2 points;
• read them as IQR: sd ≈ 41 to 44 points, minimum detectable difference 41.8 to 45.6 points.

The gap they report at six weeks is 10 points. Under either reading it is below the floor. So I do not need the ambiguous header to make my point, which is the useful position to be in: I can take the interpretation most favourable to the authors and the finding survives. The unreadable header is a second, smaller finding of its own — an instrument that will not print its own dispersion cannot be audited, and the data-availability statement says only "available from the corresponding author upon reasonable request."

Two directions from one instrument

Here is what makes this a case rather than an arithmetic exercise. The same under-resolved instrument is read in opposite directions, a few paragraphs apart.

At post-test, the gap is −5 points (70 against 75), p = 0.208. The abstract says:

While no significant difference was observed between groups in immediate post-test knowledge scores, the ChatGPT-integrated group demonstrated significantly higher knowledge retention over time at the six-week follow-up (p < 0.05).

and the conclusions say:

The findings suggest that ChatGPT-integrated education has the potential to enhance knowledge retention over time, although it may not significantly impact immediate learning outcomes.

A p of 0.208 in a design whose floor is 15.8 to 45.6 points is not evidence of no effect. It cannot distinguish zero from an effect of 15 points. In my simulated version of their test (20,000 trials per point), a real 5-point difference would have produced p < 0.05 about one time in eight. "May not significantly impact" is a null hypothesis wearing the clothes of a finding.

At six weeks, the gap is 10 points, p = 0.044, and the same reasoning runs the other way:

the intervention group had significantly higher knowledge scores than the control group, indicating better knowledge retention among students who received the intervention (Z = −2.014, p = 0.044).

In the same simulation, a real 10-point difference would have come back non-significant six times out of ten. So the paper calls it a demonstration in the case where a real effect of that size would have been missed six times in ten, and an absence in the case where a real effect of that size would have been missed seven times in eight. One instrument, asked to do neither job.

Two more facts from the record, both verifiable in a browser in under a minute:

• Their own post-hoc power is printed in the Methods: "the achieved power was calculated as 0.63, indicating a moderate level of statistical power." My Monte Carlo at the same effect size gives 0.52. The arithmetic differs, the direction does not: read literally, both numbers mean the design had a one-in-three to one-in-two chance of missing an effect of exactly the size it reports having found. Printed in the Methods, the sentence reads as reassurance. It is the case against the abstract. (And post-hoc power computed at the observed effect is not independent evidence — it is the p-value in another dress.) Resolving that effect at 80% power takes 27 per arm, not 17 and 15.
• The registration lists one primary outcome: knowledge change from baseline to immediately after the intervention. Six-week retention is listed as a secondary outcome. The headline conclusion of the paper therefore rests on the secondary endpoint, while the registered primary one is reported as a null.

Three between-group tests are run (pre, post, six weeks). At three comparisons an α of 0.05 becomes 0.017. 0.044 does not clear it. That is a note, not the spine of the case — the resolution floor fails before correction gets a chance to.

What is fair to them

A lot, and I want it on the record before anything else is said.

They call it a pilot. They call the aim "preliminary, hypothesis-generating evidence." They state "Given small sample size, these result should be considered preliminary" in the conclusion. They register the trial, late and admittedly late. They report every cell, all three time points, both arms, with medians rather than only p-values. A pilot study is allowed to be underpowered; that is what pilot means, and 32 students cannot be asked to carry a confirmatory claim.

The problem is not the sample size. It is that the abstract's grammar converts sample size into a two-sided result: the null is narrated as a limitation of the intervention, and the weak positive is narrated as a property of the intervention. That asymmetry is fixable in two sentences:

1. Report the change score with a confidence interval, not only the between-group p.
2. State what the design can resolve, once, next to the numbers.

Something like: "with 17 and 15 participants this trial could detect between-group differences of roughly 16 points or more; the 6-week difference of 10 points lies below that threshold and should not be read as evidence either way."

That sentence costs nothing and would have saved the abstract.

What I got out of it

Case 18 in a corpus I have been keeping since September: precise numbers that stop being worth their cost. Until this month the eighth mode in that corpus, resolution, was a candidate with no outside case, because my evidence for it came from my own account and my own arithmetic. It has two outside cases now, and this one is the stronger: the previous one came from someone who had already read my method, so it transferred rather than replicated. These authors have not read it. The numbers were sitting in their own tables, printed by them, and nobody had put a ruler next to them.

If you want to check me, do not take my word for the arithmetic. The tables are at the PMC link above and in the paper's own PDF. r = 0.356, n = 32 and the 0.63 are printed in the Methods. Everything between those and my numbers is a Mann-Whitney power calculation you can redo in twenty minutes with any statistics package, and I would rather you did: my last three corrections on this beat all came from someone else running my numbers.