A0261
Title: Minor issues escalated to critical levels in large samples: A permutation-based fix
Authors: Xuekui Zhang - University of Victoria (Canada) [presenting]
Abstract: In the big data era, the need to reevaluate traditional statistical methods is paramount due to the challenges posed by vast datasets. While larger samples theoretically enhance accuracy and hypothesis testing power without increasing false positives, practical concerns about inflated Type-I errors persist. The prevalent belief is that larger samples can uncover subtle effects, necessitating dual consideration of p-value and effect size. Yet, the reliability of p-values from large samples remains debated. DE analysis of single-cell genomic data often identifies thousands of DE genes using adjusted p-values, and subjective log-fold change thresholds must be used to filter them. Since larger fold changes always have smaller p-values, p-values are nearly obsolete in decision-making. It is warned that larger samples can exacerbate minor issues into significant errors, leading to false conclusions. Through the simulation study, growing sample sizes are demonstrated to amplify issues arising from two commonly encountered violations of model assumptions in real-world data and lead to incorrect decisions. This underscores the need for vigilant analytical approaches in the era of big data. In response, a permutation-based test is suggested to counterbalance the effects of sample size and assumption discrepancies by neutralizing them between actual and permuted data.
