Misuses of Statistical Significance

Common misuses of statistical significance in reviewing research (meta-analysis)

In

When

Misuse

The

social

sciences

To assess whether effect are non zero

Bias toward the conclusion that the treatment has not effect when excluding studies with no significant effect as a whole (when some small effect undetected might exist).

Assess the consistency of effects across replicated studies

Magnitude effect of the studies are considered consistent if the outcomes of the significance tests are largely consistent. There is no easy way to tell whether the results of studies are consistent with one another from the outcomes of their individual significance tests alone.

The combined effects of erroneous procedures for evaluating existence and consistency of effects

If most effects are small to medium and size are moderate, then there is likely to a predominance of non significant results in the reviewed studies.

Statistical procedures for combining estimates of effect magnitude

Recent advances on statistical meta-analysis showed that procedures as analysis of variance or regression analysis cannot be justified on either statistical or conceptual ground. But scale-free indices exist. (effect size-standardized mean difference, and product moment correlation).

In conventional

statistical methodology

in research synthesis

Goals of statistical procedures in research synthesis

Should be:

Average effect can be estimated across studies,

Consistency of effect,

Effect of variables that define differences among studies,

Significance of variation across levels of the explanatory variables can be tested to determine if all variations in effect are essentially explained.

Conventional analysis for effect size data

First, calculate an effect size with Glass’s standardized difference in case of various studies and meta-analysis

Conceptual problems

It is impossible to directly test the consistency of effect sizes across studies (or wether the systematic variation in k effect sizes is larger than the nonsystematic exhibited by those effects sizes).

If the investigator tries to “explain” variation by grouping studies with similar characteristics there is no way to assess whether the remaining variation among effect sizes is systematic or random.

Statistical problems with conventional analysis

Homoskedasticity assumption pretends the residual variance stay reasonably constant. In meta-analysis, with different size studies error variance so heterogeneous. In case of research synthesis this heterogeneity can be high.

Improper modeling of initial observations

Ignore time persistent errors that are correlated with lag dependent variable

In Cluster Analysis over time

Source: nfm