Common misuses of statistical significance in reviewing research (metaanalysis) 

In 
When 
Misuse 
The social sciences 
To assess whether effect are non zero 
Bias toward the conclusion that the treatment has not effect when excluding studies with no significant effect as a whole (when some small effect undetected might exist). 
Assess the consistency of effects across replicated studies 
Magnitude effect of the studies are considered consistent if the outcomes of the significance tests are largely consistent. There is no easy way to tell whether the results of studies are consistent with one another from the outcomes of their individual significance tests alone. 

The combined effects of erroneous procedures for evaluating existence and consistency of effects 
If most effects are small to medium and size are moderate, then there is likely to a predominance of non significant results in the reviewed studies. 

Statistical procedures for combining estimates of effect magnitude 
Recent advances on statistical metaanalysis showed that procedures as analysis of variance or regression analysis cannot be justified on either statistical or conceptual ground. But scalefree indices exist. (effect sizestandardized mean difference, and product moment correlation). 

In conventional statistical methodology in research synthesis 
Goals of statistical procedures in research synthesis 
Should be: Average effect can be estimated across studies, Consistency of effect, Effect of variables that define differences among studies, Significance of variation across levels of the explanatory variables can be tested to determine if all variations in effect are essentially explained. 
Conventional analysis for effect size data 
First, calculate an effect size with Glass’s standardized difference in case of various studies and metaanalysis 

Conceptual problems 
It is impossible to directly test the consistency of effect sizes across studies (or wether the systematic variation in k effect sizes is larger than the nonsystematic exhibited by those effects sizes). If the investigator tries to “explain” variation by grouping studies with similar characteristics there is no way to assess whether the remaining variation among effect sizes is systematic or random. 

Statistical problems with conventional analysis 
Homoskedasticity assumption pretends the residual variance stay reasonably constant. In metaanalysis, with different size studies error variance so heterogeneous. In case of research synthesis this heterogeneity can be high. 

Improper modeling of initial observations 


Ignore time persistent errors that are correlated with lag dependent variable 
In Cluster Analysis over time 

Source: nfm 