How to efficiently calculate a running standard deviation

by

The answer is to use Welford’s algorithm, which is very clearly defined after the “naive methods” in:

It’s more numerically stable than either the two-pass or online simple sum of squares collectors suggested in other responses. The stability only really matters when you have lots of values that are close to each other as they lead to what is known as “catastrophic cancellation” in the floating point literature.

You might also want to brush up on the difference between dividing by the number of samples (N) and N-1 in the variance calculation (squared deviation). Dividing by N-1 leads to an unbiased estimate of variance from the sample, whereas dividing by N on average underestimates variance (because it doesn’t take into account the variance between the sample mean and the true mean).

I wrote two blog entries on the topic which go into more details, including how to delete previous values online:

You can also take a look at my Java implement; the javadoc, source, and unit tests are all online:

More Related Contents:

  1. Convert perl regex to python [closed]
  2. Randomly generate 4 or 5 digit Unique Codes in Subsequent Runs of the Script
  3. How can I remove text within parentheses with a regex?
  4. Find p-value (significance) in scikit-learn LinearRegression
  5. Compute a confidence interval from sample data
  6. Calculating Pearson correlation and significance in Python
  7. Constructing a co-occurrence matrix in python pandas
  8. How to highlight specific x-value ranges
  9. How to plot empirical cdf (ecdf)
  10. What do all the distributions available in scipy.stats look like?
  11. confidence and prediction intervals with StatsModels
  12. Scraping dynamic content in a website [duplicate]
  13. What are the differences between Perl, Python, AWK and sed? [closed]
  14. Pythonic way of detecting outliers in one dimensional observation data
  15. Pandas – Compute z-score for all columns
  16. scipy, lognormal distribution – parameters
  17. How to incrementally sample without replacement?
  18. How do I calculate percentiles with python/numpy?
  19. How do I calculate r-squared using Python and Numpy?
  20. How to calculate cumulative normal distribution?
  21. How to plot ROC curve in Python
  22. Boxplots in matplotlib: Markers and outliers
  23. T-test in Pandas
  24. Character Translation using Python (like the tr command)
  25. plotting a histogram on a Log scale with Matplotlib
  26. Multivariate polynomial regression with numpy
  27. Real-time intercepting of stdout from another process in Python
  28. Compute a confidence interval from sample data assuming unknown distribution
  29. Regexp finding longest common prefix of two strings
  30. Computing cross-correlation function?

Leave a Comment