Statistics for Data Science — Part I: Introduction
In 1970s, John Tukey produced a new definition for statistics; instead of calling it a pure mathematical science, he suggested that deriving hypotheses from data was the future. It was a reform of statistics and announcement of an as-yet unrecognized science. It has been called Data Science for a long time and it is influenced by computer science, mathematics, statistics as well as the applied sciences.
In this series of articles, I will cover the basic parts of statistics which are crucial for a data scientist and I will try to answer the following questions:
· What is statistics?
· Why should I learn statistics?
· How can statistics help me in my profession?
Definitions and Concepts
For the sake of formality, let us start with the Wikipedia definition for statistics:
”Statistics is the discipline that concerns the collection, organization, analysis, interpretation, and presentation of data.”
There are two basic types of data sets, namely, a population and a sample.
A population is the collection of all outcomes, responses, measurements, or counts that are of interest.
Meanwhile, a sample is a subset of a population. Sample data are used to form conclusions about populations.
The figure below shows an example of systematic sampling, where the sample set is generated by picking specimens by following a certain pattern (not randomly).
A parameter is a numerical description of a population characteristic.
A statistic is a numerical description of a sample characteristic.
Descriptive and Inferential Statistics are defined as the two branches of statistics.
Descriptive statistics is the branch that involves the organization, summarization and display of data — data analysis and visualization.
Inferential statistics is the branch of statistics that involves using a sample to draw conclusions about a population. Probability is widely used in inferential statistics.
Data sets contain two types of data: qualitative and quantitative data.
Qualitative data consist of attributes, labels, or nonnumerical entries.
Quantitative data consist of numerical measurements or counts.
Frequency Distributions
A frequency distribution is a table that shows classes or intervals of data entries with a count of the number of entries in each class. The frequency f of a class is the number of data entries (or how often something happened) in the class.
In the frequency distribution shown, there are nine classes. Each class has a lower and an upper-class limit. The difference between the maximum and minimum data entries is called the range.
Sometimes, it is easier to identify patterns of a data set by looking at a graph of the frequency distribution. Frequency histogram is a very explanatory tool for the pattern illustration of data.
A frequency histogram is a bar graph that represents the frequency distribution of a data set. A histogram has the following properties:
· The horizontal axis includes the data values,
· The vertical axis includes the frequencies of the classes.
A frequency polygon shows the frequency distribution in an alternative way. It is a line graph that emphasizes the continuous change in frequencies. The difference is shown below, for the same data set.
Central Tendency
A measure of central tendency is a value that represents a typical, or central entry of a data set.
The three most commonly used measures of central tendency are the mean, the median and the mode.
The mean of a data set is the sum of the data entries divided by the number of entries.
Population Mean: Sample Mean:
where, µ is the population mean and is the sample mean.
The median of a data set is the value that lies in the middle of the data when the data is ordered. It may be considered as the middle value of a data set.
The mode of a data set is the data entry that occurs with the greatest frequency. If two entries have the same frequency, each one of them is a mode and the data set is called bimodal.
Each of these values describe a typical property of a data set and there are advantages and disadvantages of using each of them. Most often, the mean is a reliable measure because it takes every element of the data set into account — unless there are outliers. An outlier is a value which is far removed from the other elements in the data set. A data set can have one or more outliers, causing gaps in an otherwise regular distribution. The most important result is, conclusions that are drawn from a data set that contains outliers may be flawed.
Boxplots and other visualization techniques are very useful in detecting outliers and we will discuss them later in the article.
Shape of Distributions
Graphics may reveal several characteristics of a frequency distribution — shape of the distribution is one of them.
A frequency distribution is symmetric if a virtual vertical line through the middle of the distribution slices the shape into identical mirror images.
A frequency distribution is uniform when all entries have approximately equal frequencies. A uniform distribution is also symmetric.
A frequency distribution is skewed if the tail of the graph elongates more to one side than to the other.
When a distribution is symmetric and unimodal, the mean, median and mode are equal. If a distribution is skewed left, the mean is less than the median and the median is usually less than the mode. If a distribution is skewed right, the mean is greater than the median and the median is usually greater than the mode.
There may be more different shapes of distributions. In some cases, the shape cannot be classified as symmetric, uniform or skewed. A distribution can have several gaps caused by outliers, or clusters of data. Clusters usually occur when several types of data are included in one data set.
The table shows which measure to consider according to the shape of distribution.
Measures of Variation
Range is the simplest measure of variation of a data set. The range is the difference between the maximum and minimum data entries in the set (for quantitative data).
Using the range to measure the variation has a major drawback: it uses only two entries from the data set and does not give much indication of the spread of observations about the mean. Using all the values in the data set will give a more reliable value.
The deviation of an entry x in a data set is the difference between the entry and the mean.
The variance of a data set of N entries is:
The standard deviation of a data set of N entries is the square foot of the variance.
Standard deviation is usually interpreted as how measurements for a group are spread out from the average value. It is also defined as measure of the typical amount an entry deviates from the mean. The figure shows an example of two sample populations with the same mean and different standard deviations. Both populations have a mean of 100, but the red population’s standard deviation is 10; whereas the blue population has a standard deviation of 50. The result is a very distributed data for the blue data set.
Empirical Rule (68–95–99.7 Rule)
For data with a bell-shaped distribution, the standard deviation has the following characteristics:
· About 68 % of the data lie within one standard deviation of the mean.
· About 95 % of the data lie within two standard deviations of the mean.
· About 99.7 % of the data lie within three standard deviations of the mean.
Measures of Position
The three quartiles, Q1, Q2 and Q3, approximately divide an ordered data set into four equal parts. About one quarter of the data fall on or below the first quartile Q1. About one half of the data fall on or below the second quartile Q2 (the second quartile is the same as the median of the data set). About three quarters of the data fall on or below the third quartile Q3.
The interquartile range (IQR) of a data set is the difference between the third and first quartiles.
IQR is a measure of variation that gives an idea of how much the middle 50 % of the data varies. It can also be used to identify the outliers. Any data value that lies more than 1.5 IQRs to the left of Q1 or to the right of Q3 is an outlier.
A box-and-whisker plot (shown below) is an exploratory data analysis tool that highlights the important features of a data set.
Next article will cover probability-related concepts.
You can access to this article and similar ones here.
