Two rigor gaps in the vignette, both aimed at the applied and clinical audience the package targets:
-
Pre-specification. The margins used (the 3-unit bar in Showcase 1 and the 20% rule in Showcase 3) read as chosen for the examples. For a superiority or threshold test, the margin has to come from domain knowledge and be fixed before seeing the estimate, not selected afterward. Add one sentence making that explicit, since a margin chosen after the estimate is in view is exactly what this audience is trained to distrust.
-
Dose composition of the dynamic threshold. In Showcase 3, overall_avg is computed from the full dataset across all three doses, so the threshold shifts with the experiment's dose composition, not only with sampling noise. A design weighted toward higher doses would move the benchmark. This is a separate concern from the estimation-uncertainty footnote already added (which covers treating a data-derived threshold as fixed), and deserves its own clause.
Location: vignettes/usage/beyond-null.Rmd, Showcases 1 and 3.
Two rigor gaps in the vignette, both aimed at the applied and clinical audience the package targets:
Pre-specification. The margins used (the 3-unit bar in Showcase 1 and the 20% rule in Showcase 3) read as chosen for the examples. For a superiority or threshold test, the margin has to come from domain knowledge and be fixed before seeing the estimate, not selected afterward. Add one sentence making that explicit, since a margin chosen after the estimate is in view is exactly what this audience is trained to distrust.
Dose composition of the dynamic threshold. In Showcase 3,
overall_avgis computed from the full dataset across all three doses, so the threshold shifts with the experiment's dose composition, not only with sampling noise. A design weighted toward higher doses would move the benchmark. This is a separate concern from the estimation-uncertainty footnote already added (which covers treating a data-derived threshold as fixed), and deserves its own clause.Location:
vignettes/usage/beyond-null.Rmd, Showcases 1 and 3.