Background: Meta-analyses conventionally report pooled effect sizes, confidence intervals, and p-values. These measures address statistical significance and precision, but not how far the results are from a “no benefit” (nb) point, as measured on a standardized robustness scale. The Neutrality Boundary Framework provides a robust metric, nb, that ranges from 0 to 1 for individual studies, but its use in meta-analysis has not been formally described. A nb of 0 indicates that results are at the neutrality boundary, where no relationship between treatment and outcome is present; a nb of 1 indicates that results show a strong relationship between treatment and outcome.
Methodology: A sample of published clinical trials was selected for evaluation. The distribution of nb across all trials was summarized, and the correlation between nb and p-values was examined.
Results: Across 161 trials, the median nb was 0.147 (range 0.000 to 0.902). Using standardized robustness bands derived from prior validation work based on simulation and trial data, 35.4% of trials showed weak robustness (nb less than 0.075), 24.8% moderate robustness (nb between 0.075 and 0.227), and 39.8% strong robustness (nb greater than or equal to 0.227). Trials with binary 2-by-2 outcomes tended to have lower nb values than trials with continuous outcomes. The robust metric nb was moderately correlated with p-values (r = 0.35, p less than 0.001). Although a moderate level of correlation was present, p-values explained only about 12% of the variation in nb (r-squared = 0.12), indicating that nb captures substantial additional information about distance from neutrality that p-values alone do not provide.
Conclusion: The distribution of trial-level nb values provides a simple way to describe robustness across a collection of studies and offers a cross-design view of how far the overall evidence base lies from therapeutic neutrality. Incorporating robust measures alongside p-values in routine evidence synthesis may provide a more complete picture of the strength and reliability of the evidence.

Meta-analysis, Neutrality boundary framework, Statistical robustness, Evidence synthesis, Statistical fragility, Complete statistical evidence.

Meta-analysis is the dominant tool for quantitative evidence synthesis in clinical research.[1] Standard practice focuses on three primary outputs: a pooled effect estimate (e.g., risk ratio, hazard ratio, mean difference), a confidence interval quantifying uncertainty, and a hypothesis test p-value against a null hypothesis of no effect.[2] Together, these answers: “Is there statistically detectable evidence against the null, and what is the estimated magnitude and precision of that effect?” However, p-values are inherently threshold-dependent, defined by an arbitrary cutoff (typically alpha=0.05).[3] This creates a fundamental problem: two meta-analyses with p-values of 0.049 and 0.001 are both labeled “statistically significant,” yet they represent vastly different levels of evidence quality. The first result sits precariously at the significance boundary and could easily flip with minor changes to the data, while the second shows robust separation from the null hypothesis. Current meta-analytic practice provides no standardized method for distinguishing between these scenarios or for quantifying how far pooled results lie from therapeutic neutrality on a scale comparable across different study designs and outcome types.

Several approaches have attempted to address the limitations of p-value-only reporting in meta-analysis. Confidence intervals provide information about precision and the magnitude of the effect. Prediction intervals estimate the range of effects in future studies.[4] Heterogeneity statistics (I-squared, tau-squared) quantify variability across studies. GRADE assessments evaluate overall evidence quality through multiple dimensions.[5] More recently, fragility indices have been extended to meta-analysis to measure how many outcome changes would be needed to flip statistical significance.[6] Each of these methods addresses essential aspects of evidence quality and has contributed to a more nuanced interpretation of meta-analytic results.

However, these existing approaches fall short in one critical dimension: none provides a threshold-free, standardized measure of how far pooled results are from therapeutic neutrality that works consistently across diverse study designs. Confidence intervals depend on arbitrary significance levels.[7] Prediction intervals estimate the range of true effects in future studies and can indicate whether those effects might be null or beneficial.[8,9] However, prediction intervals do not provide a standardized measure of how far the pooled effect lies from neutrality. The neutrality boundary framework (NBF) addresses this gap by quantifying distance from neutrality on a standardized 0-1 scale. Fragility indices are, by definition, threshold-dependent, measuring only changes needed to cross the alpha=0.05 boundary. GRADE assessments, while comprehensive, do not quantify the extent to which the results deviate from no effect on a standardized 0–1 scale. What remains absent is a metric that answers the question: “Regardless of whether p is above or below 0.05, how far are these results from a ‘no effect’ point, measured consistently across binary, continuous, survival, and other outcome types?”

NBF addresses this gap by providing a robustness index, nb, that ranges from 0 to 1 and measures geometric distance from therapeutic neutrality.[10] The NBF has been successfully developed and applied to individual clinical trials across multiple study designs, including independent binary outcome trials, matched binary outcome trials, diagnostic test evaluations, continuous outcomes, correlation analyses, and survival analyses.[11]

The NBF defines robustness using the general formula: nb = |T – T0| / (|T – T0| + S), where T is a test statistic or effect measure (for example, a log risk ratio or a standardized mean difference), T0 is the point of therapeutic neutrality (for example, log risk ratio = 0 or correlation r = 0), and S is a scale parameter that reflects within-study variability (for example, a standard error or a pooled standard deviation). The metric is threshold-free (independent of any chosen significance level), design-agnostic (works across different study types on a standardized 0-1 scale), and interpretable (values near 0 indicate proximity to neutrality; higher values indicate stronger separation from neutrality). However, its extension to meta-analytic evidence synthesis has not been formalized. Meta-analyses currently lack a standardized method for applying NBF principles when synthesizing evidence across studies with varying designs, sample sizes, and effect sizes.

The objective of this study was to extend the NBF to meta-analysis through two complementary approaches. The first approach, implemented empirically in this paper, involves summarizing trial-level nb distributions across heterogeneous study designs to characterize the overall robustness of an evidence base. The second approach, formalized theoretically, defines a pooled robustness measure (nb_meta) for future use when study-level effect estimates and their variances are available. Together, these approaches provide meta-analysts and clinicians with a standardized method to assess how far synthesized evidence lies from therapeutic neutrality, complementing traditional p-values and addressing the critical gap in current meta-analytic practice.

Study design: This study used two complementary approaches to assess robustness in meta-analysis using the NBF. The first approach involved descriptive synthesis of trial-level nb values across a heterogeneous sample of clinical trials. The second approach formalized a pooled robustness measure (nb_meta) for future implementation when study-level effect estimates and variances are available.

Data source: The analysis used a convenience sample of 161 clinical trials for which NBF analyses had already been completed. For each trial, the following data were available: DOI or PubMed identifier, publication year, clinical specialty, study design category, p-value, and NBF robustness metric nb. Trials were classified into six design categories: binary 2-by-2 independent, binary 2-by-2 matched, continuous-difference, continuous-means, survival outcomes, and contingency tables. Trial-level nb values had been computed using design-specific NBF methods appropriate for each study type. As the majority of trials were two-arm, independent binary outcome trials, the NBF metric utilized was the Risk Quotient (RQ). For 2×2 contingency tables, the RQ = |ad – bc| / (N^2 / 4), which quantifies how far results are from independence (the point where treatment and outcome are unrelated, i.e., a relative risk of one).[12] All nb values range from 0 to 1, where values near 0 indicate proximity to therapeutic neutrality and values near 1 indicate strong separation from neutrality.

Meta-analysis robustness framework: When study-level effect estimates are available, a pooled robustness metric can be defined using NBF principles. The pooled robustness measure nb_meta is defined as:

nb_meta = |T_pooled| / (|T_pooled| + SE_pooled)

Here, T_pooled is the pooled effect estimate expressed on a neutrality-centered scale (neutrality at T = 0), and SE_pooled is the standard error of the pooled estimate under the specified meta-analytic model (fixed-effects or random-effects).

This definition yields values between 0 and 1. When the pooled effect is large relative to its pooled uncertainty, nb_meta-approaches 1, indicating strong separation from therapeutic neutrality across the evidence base. When the pooled effect is small relative to pooled uncertainty, nb_meta-approaches 0, indicating proximity to neutrality. Computing nb_meta requires individual-study effect estimates and standard errors (or equivalent variance information) so that T_pooled and SE_pooled can be obtained from a standard meta-analysis.

To demonstrate the practical application of nb_meta, a published diagnostic meta-analysis of cadmium-zinc-telluride (CZT) SPECT for coronary artery disease detection was selected as a worked example.[13] This meta-analysis included 11 studies with complete 2-by-2 diagnostic accuracy data, enabling calculation of individual study robustness using the Diagnostic Neutrality Boundary metric and subsequent pooling using the nb_meta formula described above.

Statistical analysis: The distribution of nb across all 161 trials was summarized using the median, range, and mean. Trials were categorized into robustness bands based on empirically derived thresholds from prior validation work: weak robustness (nb<0.075), moderate robustness (0.075≤ nb<0.227), and strong robustness (nb≥0.227). The number and proportion of trials in each band were calculated. The Pearson correlation coefficient between nb and -log10(p) was computed to examine the relationship between robustness and statistical significance. The -log10(p) transformation converts p-values to a scale where larger values indicate stronger evidence against the null hypothesis. Trials with p-values reported as zero were excluded from correlation analysis. SAMPL guidelines were followed.[14]

Ethics statement: This study is a secondary analysis of aggregate statistical results reported in previously published medical literature. No individual-level data, identifiable private information, or identifiable biospecimens were obtained, accessed, or analyzed. Under 45 CFR 46, this project does not constitute human subjects research, and institutional review board oversight was not required.

Dataset overview: The dataset contained 161 clinical trials with complete p-value and nb information. By design class, binary 2-by-2 independent trials comprised the majority (n=133, 82.6%), followed by continuous-difference (n=15, 9.3%), continuous-means (n=7, 4.3%), survival outcomes (n=3, 1.9%), contingency tables (n=2, 1.2%), and binary 2-by-2 matched (n=1, 0.6%). Trials spanned multiple specialties, including autoimmunity, cardiology, endocrinology, gastroenterology, infectious diseases, internal medicine, nephrology, neurology, oncology, pediatrics, psychiatry, public health, pulmonary medicine, and surgery.

Overall robustness distribution: Across all 161 trials, the robustness metric nb had a median of 0.147 (range 0.000 to 0.902) and a mean of 0.247. By robustness band, 57 trials (35.4%) showed weak robustness, 40 trials (24.8%) showed moderate robustness, and 64 trials (39.8%) showed strong robustness. Overall, most trials exhibited weak or moderate robustness, indicating that their estimated effects remained relatively close to therapeutic neutrality. Table 1 summarizes these statistics.

Table 1: Overall distribution of robustness metric nb across all trials

Relationship between robustness and p-values: The empirical relationship between robustness (nb) and p-values was examined using -log10(p) as a measure of p-value strength. The correlation coefficient between nb and -log10(p) was r=0.35 (Pearson, p<0.001, n=137). Twenty-four trials with p-values reported as zero or below machine precision were excluded from this correlation analysis. This pattern shows that robustness, as quantified by nb, is related to, but not reducible to, statistical significance. Only 12% of the variation in nb is explained by the p-value (r-squared = 12.2%). This confirms that the geometric distance from neutrality (nb) is quantitatively different from the raw incompatibility of the data with a null hypothesis (p-value). Continuous-outcome designs tended to show higher robustness (larger nb) than binary designs, consistent with higher information per subject in continuous endpoints.

Empirical validation of pooled robustness: To demonstrate the practical implementation of the proposed nb_meta pooling metric, the Neutrality Boundary Framework was applied to a published diagnostic meta-analysis evaluating cadmium-zinc-telluride (CZT) SPECT for coronary artery disease detection.[13] This meta-analysis included 11 studies with complete 2×2 diagnostic accuracy data, enabling calculation of individual study robustness (nb) via the Diagnostic Neutrality Boundary (DNB) metric and subsequent pooling.[15-25] Because all cells were nonzero, DOR, SE[ln(DOR)], and DNB were calculated directly from the 2×2 counts without continuity corrections.

Individual study robustness calculations: For each study, the diagnostic odds ratio (DOR) was computed from the reconstructed 2×2 table:

DOR = (TP × TN) / (FP × FN)

The DNB metric was then calculated as:

DNB = |ln(DOR)| / (|ln(DOR)| + SE[ln(DOR)])

where SE[ln(DOR)] = √(1/TP + 1/FN + 1/FP + 1/TN)

The DOR and DNB results are shown in Table 2.

Table 2: Diagnostic accuracy data and individual study robustness for 11 CZT SPECT studies.

Legend: The 2-by-2 diagnostic accuracy data (TP, FP, TN, FN) for each study were used to calculate the diagnostic odds ratio (DOR), its natural log and standard error, and the Diagnostic Neutrality Boundary (DNB), which is the NBF robustness metric for diagnostic studies. TP = true positives; FP = false positives; TN = true negatives; FN = false negatives; SE = standard error of ln(DOR); DNB = nb for diagnostic studies.

Pooled robustness calculation: Following the NBF formula for pooled robustness, nb_meta was calculated in two steps. First, the pooled effect estimate was obtained by weighing each study’s ln(DOR) by its precision, with more precise studies (those with smaller standard errors) receiving greater weight. The weight for each study was calculated as 1 divided by the square of its standard error. Second, the pooled effect estimate and its corresponding standard error were entered into the NBF formula to obtain nb_meta = (pooled effect) / (pooled effect + pooled standard error). The pooled effect calculation is shown in Table 3.

Table 3: Inverse-variance weighted pooling of ln(DOR)

Legend: Each study’s ln(DOR) was weighted by 1 divided by SE squared. The total weight sum (20.492) and weighted ln(DOR) sum (52.826) were used to calculate the pooled effect (52.826/20.492 = 2.578) and the pooled standard error (1/sqrt(20.492)=0.221). These are used to obtain nb_meta = 2.578 / (2.578 + 0.221) = 0.921. DOR = diagnostic odds ratio, SE = standard error of ln(DOR).

Interpretation: The pooled robustness of nb_meta = 0.92 indicates that the CZT SPECT diagnostic evidence is strongly separated from therapeutic neutrality (DOR = 1), consistent with the meta-analysis conclusion that CZT SPECT demonstrates reliable diagnostic accuracy for coronary artery disease. On the 0–1 NBF scale, values approaching 1 reflect maximal separation from neutrality; the observed value of 0.92 indicates that the pooled effect estimate is large relative to its uncertainty.

Notably, one study exhibited substantially lower robustness (nb = 0.29), reflecting its small sample size (n = 21) and a near-neutral diagnostic odds ratio (DOR = 1.75) [21]. The inverse-variance weighting appropriately down-weights this imprecise estimate in the pooled ln(DOR), demonstrating a key advantage of the nb_meta approach: studies contributing weak or unstable evidence receive proportionally less influence on the pooled estimate.

This empirical demonstration confirms that nb_meta can be computed directly from published meta-analysis forest plots without requiring access to individual patient data, supporting its feasibility for routine implementation in systematic reviews.

The central finding of this study is that robustness, as quantified by the NBF metric nb, can be synthesized across heterogeneous clinical trials to provide a standardized assessment of how far an evidence base lies from therapeutic neutrality. Across 161 trials spanning multiple study designs and clinical specialties, the median nb was 0.147, with most trials (60.2%) exhibiting weak or moderate robustness. This indicates that most trial results remained relatively close to the point of no effect, even when statistically significant. The moderate correlation between nb and p-values (r=0.35) demonstrates that robustness and statistical significance measure related but distinct dimensions of evidence quality. Only 12% of the variation in nb is explained by p-values, confirming that robustness provides substantial additional information beyond what p-values alone convey.

This study makes three contributions to meta-analytic methodology. First, it demonstrates empirical robustness synthesis using trial-level nb distributions across heterogeneous study designs, showing that robustness can be meaningfully summarized without requiring uniform outcome types or effect size metrics. Second, it formalizes the theoretical framework for pooled meta-analytic robustness (nb_meta in the methods section) for future implementation once study-level effect estimates and variances become available. Third, it establishes robustness as a complementary dimension of meta-analytic evidence alongside pooled p-values, providing a threshold-free assessment of evidence quality that works consistently across binary, continuous, survival, and other outcome types.

A common objection is that confidence intervals already contain the necessary information about distance from the no-effect point. Confidence intervals indicate whether the interval includes the null and how wide the compatible range of effects is, so they are a good measure of precision. However, they do not provide a standardized 0-to-1 summary of the distance from neutrality relative to its uncertainty, and they are not directly comparable across different effect scales and study designs. In contrast, nb and nb_meta take the same basic ingredients (effect estimate and its variability) and convert them into a single robustness score between 0 and 1. This score reflects both the distance of the estimate from the no-effect value and its noise, on a common scale that is interpretable and directly comparable across diverse designs. In this sense, nb and nb_meta do not duplicate the confidence interval; they extract and standardize the distance-from-neutrality information that the interval only displays in a plot or on the original effect scale.

Classical meta-analyses emphasize pooled effects, confidence intervals, and p-values, sometimes supplemented by prediction intervals or GRADE assessments.[26] Fragility indices for meta-analysis quantify how many outcome changes would flip pooled significance. NBF robustness differs in three fundamental ways. It is threshold-free, remaining independent of the arbitrary alpha=0.05 cutoff that defines statistical significance. It is explicitly geometric, measuring distance from neutrality relative to variability rather than probability under a null hypothesis. It is cross-design, providing a unified 0-to-1 scale across binary, continuous, survival, and other outcome types where traditional effect sizes are not directly comparable. By integrating nb alongside p-values, meta-analysts can complement significance testing with a standardized robustness layer that answers a fundamentally different question: how far are these results from no effect?

In practice, meta-analysts could report the median nb and proportion of trials exceeding robustness thresholds alongside pooled effects. They could examine design-specific or domain-specific robustness patterns to identify whether certain study types consistently show stronger or weaker separation from neutrality. Systematically low nb values (e.g., nb < 0.227) would indicate proximity to neutrality even when p < 0.05, suggesting that statistically significant findings may lack practical separation from no effect. Consistently high nb values (e.g., nb > 0.227) would indicate robust separation from neutrality, strengthening confidence in the magnitude of the observed effects. This approach overlays a robustness map atop standard meta-analytic outputs without replacing them, providing complementary information for interpreting evidence and making clinical decisions.

Limitations: This analysis used a convenient sample of trials with completed NBF analyses, assembled for methodological illustration rather than systematic review of a specific clinical question. Results demonstrate feasibility and interpretive principles but should not be interpreted as clinical evidence for any intervention. The sample was heterogeneous by design, spanning multiple clinical specialties and study types, which enabled demonstration of cross-design robustness synthesis but limits generalizability to any single clinical domain. Directionality (benefit versus harm) was not emphasized in this analysis. The focus was on absolute distance from neutrality. However, in applied settings, nb can be paired with the sign of the underlying effect to distinguish beneficial from harmful deviations from null. The pooled robustness measure nb_meta was introduced as a conceptual extension of NBF to meta-analytic effect sizes, but empirical validation and software implementation are left for future work. The weak, moderate, and strong robustness bands are based on empirically derived provisional recommendations from 118 real trials and 1 million simulated trials. The NBF formula structure supports the use of standard thresholds across designs by placing all robustness metrics on a shared 0-to-1 scale. However, these bands remain provisional and warrant further validation in specific clinical domains. Finally, the observed correlation between nb and p-values is descriptive. Patterns may differ across datasets, particularly when distributions of study designs, sample sizes, or effect magnitudes differ.

Future work should focus on implementing nb_meta using extracted effect sizes from published meta-analyses, validating robustness thresholds in domain-specific contexts, and developing software tools to facilitate routine calculation and reporting of robustness metrics alongside traditional meta-analytic outputs. As robustness assessment becomes integrated into standard evidence synthesis workflows, the paucity of threshold-free, cross-design metrics in current practice will become increasingly apparent, and the value of complementing p-values with geometric distance measures will become more widely recognized.

Neutrality boundary robustness provides a simple, interpretable, threshold-free measure of distance from therapeutic neutrality. This study demonstrates that nb extends naturally to meta-analytic evidence through descriptive synthesis of trial-level nb distributions and the definition and worked example of pooled nb_meta based on meta-analytic effects and cross-study dispersion. Confidence intervals and p-values remain essential, but they do not, by themselves, provide a standardized, cross-trial 0-to-1 measure of how far results lie from the no-effect point. The NBF fills this gap by translating effect size and uncertainty into a common robustness scale.

Integrating robust metrics alongside p-values and confidence intervals yields a more complete picture of evidential strength. The Neutrality Boundary Framework, extended to meta-analysis, offers a threshold-free, design-agnostic approach to assessing evidence quality across individual trials and synthesized bodies of evidence, helping clinicians distinguish results that merely cross a significance threshold from those that are truly far from neutrality.

Not reported

No external funding was received.

Thomas F. Heston
Department of Family Medicine, University of Washington, USA
Department of Medical Education and Clinical Sciences, Washington State University, USA
Email: tomhestonmd@gmail.com

The author independently designed the study; collected, analyzed, and interpreted the data; drafted and critically revised the manuscript; approved the final version for publication; and accepts full responsibility for all aspects of the work.

Not applicable

The author declares no conflicts of interest.

None

https://doi.org/10.63096/medtigo3062417

Heston TF. Neutrality Boundary Robustness for Meta-Analysis. medtigo J Med. 2026;4(1):e3062417. doi:10.63096/medtigo3062417 Crossref