Seeing if data is normally distributed in R

by

Normality tests don’t do what most think they do. Shapiro’s test, Anderson Darling, and others are null hypothesis tests AGAINST the the assumption of normality. These should not be used to determine whether to use normal theory statistical procedures. In fact they are of virtually no value to the data analyst. Under what conditions are we interested in rejecting the null hypothesis that the data are normally distributed? I have never come across a situation where a normal test is the right thing to do. When the sample size is small, even big departures from normality are not detected, and when your sample size is large, even the smallest deviation from normality will lead to a rejected null.

For example:

> set.seed(100)
> x <- rbinom(15,5,.6)
> shapiro.test(x)

    Shapiro-Wilk normality test

data:  x 
W = 0.8816, p-value = 0.0502

> x <- rlnorm(20,0,.4)
> shapiro.test(x)

    Shapiro-Wilk normality test

data:  x 
W = 0.9405, p-value = 0.2453

So, in both these cases (binomial and lognormal variates) the p-value is > 0.05 causing a failure to reject the null (that the data are normal). Does this mean we are to conclude that the data are normal? (hint: the answer is no). Failure to reject is not the same thing as accepting. This is hypothesis testing 101.

But what about larger sample sizes? Let’s take the case where there the distribution is very nearly normal.

> library(nortest)
> x <- rt(500000,200)
> ad.test(x)

    Anderson-Darling normality test

data:  x 
A = 1.1003, p-value = 0.006975

> qqnorm(x)

enter image description here
enter image description here

Here we are using a t-distribution with 200 degrees of freedom. The qq-plot shows the distribution is closer to normal than any distribution you are likely to see in the real world, but the test rejects normality with a very high degree of confidence.

Does the significant test against normality mean that we should not use normal theory statistics in this case? (another hint: the answer is no 🙂 )

More Related Contents:

  1. Gradient of n colors ranging from color 1 and color 2
  2. Unable to install packages in latest version of RStudio and R Version.3.1.1 [duplicate]
  3. Check if the number is integer
  4. How can I make consistent-width plots in ggplot (with legends)?
  5. R.exe, Rcmd.exe, Rscript.exe and Rterm.exe: what’s the difference?
  6. How to get the maximum value by group
  7. Specification of first and last tick marks with scale_x_date
  8. R list of lists to data.frame
  9. How to draw stacked bars in ggplot2 that show percentages based on group?
  10. Is it possible to use spread on multiple columns in tidyr similar to dcast? [duplicate]
  11. R – image of a pixel matrix?
  12. Is it possible to stop executing of R code inside shiny (without stopping the shiny process)?
  13. Returning anonymous functions from lapply – what is going wrong?
  14. get_map not passing the API key (HTTP status was ‘403 Forbidden’)
  15. Euclidean distance of two vectors
  16. Get rid of \addlinespace in kable
  17. facet_wrap add geom_hline
  18. Splitting a data frame into equal parts
  19. Deploying R shiny app as a standalone application [closed]
  20. Set one or more of coefficients to a specific integer
  21. Bars in geom_bar have unwanted different widths when using facet_wrap
  22. Generate correlated random numbers from binomial distributions
  23. In R data.table, how do I pass variable parameters to an expression?
  24. R Knitr PDF: Is there a posssibility to automatically save PDF reports (generated from .Rmd) through a loop?
  25. Connect stack bar charts with multiple groups with lines or segments using ggplot 2
  26. Replicate each row of data.frame and specify the number of replications for each row?
  27. How to replace certain values in a specific rows and columns with NA in R?
  28. Make a file writable in order to add new packages
  29. Quantmod Error ‘cannot open URL’
  30. paste two data.table columns

Leave a Comment