Statistical distributions




In the previous chapter we discussed probability theory, which we expressed in terms of a variable $X$. We defined $X$ as a set of realizations of some process, which in turn is governed by rules of probability regarding potential outcomes in the sample space.

The variables we were talking about have been what are called random variables, which means that they have a probability distribution. As we noted before, broadly speaking, there are two kinds of random variables: discrete and continuous.

Discrete variables can take on any one of several distinct, mutually-exclusive values.

  • Congressperson's ideology score {0,1,2,3...,100}
  • An individual's political affiliation (Democrat, Republican, Independent}
  • Whether or not a country is a member of European Union (true/false)

A Continuous variable can take on any value in its range.

  • Individual income
  • National population

This chapter focuses on a family of continuous distributions that are the most widely used in statistical inference, and are found in a wide variety of contexts, both applied and theoretical. The <math>Normal</math> distribution is the well-known "bell-shaped curve" that most students usually encounter first in the artificial context of academic testing, but due to a powerful result called the Central Limit Theorem, occurs in a wide variety of uncontrolled situations where the value of a random variables is determined by the average effect of a large number of random variables with any combination of distributions. The <math>\chi^{2}</math>, <math>t</math> and <math>F</math> distributions can be derived from various products of normally-distributed variables, and are used extensively in statistical inference and applied statistics, so it's useful to understand them in a bit of depth.

Need to do

Philip Schrodt 06:57, 13 July 2011 (PDT)

  • Probably need to get most of the probability chapter---which are the moment hasn't been started---written before this one. In particular, will the pdf and cdf be defined there or here?
  • Add some of the discrete distributions, particularly the binomial
  • Add the uniform?
  • Do we add---or link to on another page---the derivation of the mean and standard errors for these: that code is available in CCL on an assortment of places on the web

The Normal Distribution

We are all used to seeing normal distributions described, and to hearing that something is "normally distributed." We know that a normal distribution is "bell-shaped," and symmetrical, and probably that it has some mean and some standard deviation.

Formally, if <math>X</math> is a normally distributed variate with mean <math>\mu</math> and variance <math>\sigma^{2}</math>, then:

<math>f(x) = \frac{1}{\sigma \sqrt{2\pi}} \text{exp} \left( - \frac{(x - \mu)^{2}}{2 \sigma^{2}} \right)</math>.

We denote this <math>X \sim N(\mu,\sigma^{2})</math>, and say ``<math>X</math> is distributed normally with mean mu and variance sigma squared. The symbol <math>\phi</math> is often used as a shorthand to represent the normal density in \eqref{normalden}:

<math>X \sim \phi_{\mu, \sigma^{2}}</math>.

The corresponding normal CDF -- which is the probability of a normal random variate taking on a value less than or equal to some specified number -- is (as always) the indefinite integral of \eqref{normalden}. This has no simple closed-form solution, so we typically just write:

<math>F(x) \equiv \Phi_{\mu, \sigma^{2}}(x) = \int \phi_{\mu, \sigma^{2}} f(x) d x.</math>

Here are a bunch of normal curves


Normal cumulative distribution functions

Bases for the Normal Distribution

The most common justification for the normal distribution has its roots in the 'central limit theorem'. Consider <math>i = {1,2,...N}</math> independent, real-valued random variates $<math>X_{i}</math>$, each with finite mean $<math>\mu_{i}</math>$ and variance <math>\sigma^{2}_{i} > 0</math>. If we consider a new variable $<math>X</math>$ defined as the sum of these variables:

<math>X = \sum_{i=1}^{N} X_{i}</math>

then we know that

<math> \text{E}(X) = \sum_{i=1}^{N} \mu_{i} </math>


<math> \text{Var}(X) = \sum_{i=1}^{N} \sigma^{2}_{i} </math>

The central limit theorem states that:

<math> \underset{N \rightarrow \infty}{\lim} X = \underset{N \rightarrow \infty}{\lim} \sum_{i=1}^{N} X_{i} \overset{D}{\rightarrow} N(\cdot) </math>

where the notation <math>\overset{D}{\rightarrow}</math> indicates convergence in distribution. That is, as <math>N</math> gets sufficiently large, the distribution of the sum of <math>N</math> independent random variates with finite mean and variance will converge to a normal distribution. As such, we often think of a normal distribution as being appropriate when the observed variable <math>X</math> can take on a range of continuous values, and when the observed value of <math>X</math> can be thought of as the product of a large number of relatively small, independent ``shocks or perturbations.

Properties of the Normal Distribution

  • A normal variate <math>X</math> has support in <math>\mathfrak{R}</math>.
  • The normal is a two-parameter distribution, where <math>\mu \in (-\infty, \infty)</math> and <math>\sigma^{2} \in (0, \infty)</math>.
  • The normal distribution is always symmetrical (<math>M_{3} = 0</math>) and mesokurtic.
  • item The normal distribution is preserved under a linear transformation. That is, if <math>X \sim N(\mu,\sigma^{2})</math>, then <math>aX + b \sim N(a\mu + b, a^{2} \sigma^{2})</math>. (Why? Recall our earlier results on <math>\mu</math> and <math>\sigma^{2}</math>).

The Standard Normal Distribution

One linear transformation is especially useful:

<math> \begin{align}

b & = \frac{-\mu}{\sigma} \\
a & = \frac{1}{\sigma}

\end{align} </math>.

This yields:

<math> \begin{align}

ax + b  & \sim  N(a\mu+b, a^{2} \sigma^{2}) \\ 
        &  \sim N(0,1) 

\end{align} </math>

This is the standard normal density function. We often denote this <math>\phi(\cdot)</math>, and say that "X is distributed as standard normal." We can also get this by transforming ("standardizing") the normal variate <math>X</math>...

  • If <math>X \sim N(\mu,\sigma^{2})</math>, then <math>Z = \frac{(x - \mu)}{\sigma} \sim N(0,1)</math>.
  • The density function then reduces to:

<math> f(z) = \equiv \phi(z) = \frac{1}{\sqrt{2\pi}} \text{exp} \left[ - \frac{(z)^{2}}{2} \right] </math>

Similarly, we often write the CDF for the standard normal as <math>\Phi(\cdot)</math>.

Why do we care about the normal distribution?

The normal distribution's importance lies in its relationship to the central limit theorem. As we'll discuss at more length later, the central limit theorem means that as one's sample size increases, the distribution of sample means (or other estimates) approaches a normal distribution.

Additional points needed on the normal

Philip Schrodt 07:00, 13 July 2011 (PDT)

  • More extended discussion of the CLT, and a note that if we are dealing with a data generating process where the "error" is the average (or cumulative) effect of a large number of random variables with a variety of distributions, the CLT tells us that the net effect will be normally distributed. This, in turn, explains why linear models that assume Normally distributed error---regression and ANOVA---have proven to be so robust in practice.
    Also: should the CLT be a separate section? The basis text could be obtained from Grimstead and Snell.
    Also there are a bunch of little applets on the web that demonstrate it, though typically just from the binomial.
  • Link to a number of examples of normally distributed data...should be easy to find these on the web. E.g. the classical height. Maybe SAT scores, though these are artificially normal
  • ref to the wikipedia article; there is also a nice graphic to snag from there---introductory sidebar---which shows the standard normal
  • sidebar on the log-normal?
  • something about the bivariate normal and some nice graphics of this?
  • sidebar on the issue of fat tails and how these destroyed the economy in 2007?---there is a fairly readable Wired article on this:

The <math>\chi^{2}</math> Distribution

The chi-square (<math>\chi^{2}</math>) distribution is a one-parameter distribution defined only how positive values. If <math>Z \sim N(0,1)</math>, then <math>Z^{2} \sim \chi^{2}_{1}</math>. That is, the square of a <math>N(0,1)</math> variable is chi-squared with one degree of freedom. The fact that the square of a standard normal variate is a one-degree-of-freedom chi-square variable also explains why (e.g.) a chi-squared variate is only defined for nonnegative real numbers. If <math>W_{1},W_{2},...W_{k}</math> are all independent <math>\chi^{2}_{1}</math> variables, then <math>\sum_{i=1}^{k}W_{i} \sim \chi^{2}_{k}</math>. (The sum of <math>k</math> independent chi-squared variables is chi-squared with <math>k</math> degrees of freedom). By extension, the sum of the squares of <math>k</math> independent <math>N(0,1)</math> variables are also <math>\sim \chi^{2}_{k}</math>.

The <math>\chi^{2}</math> distribution is positively skewed, with <math>\text{E}(W) = k</math> and <math>\text{Var}(W) = 2k.</math>

Figure below presents five <math>\chi^{2}</math> densities with different values of <math>k</math>.


Need to define degrees of freedom here

Characteristics of the <math>\chi^{2}</math> Distribution

If <math>W_{j}</math> and <math>W_{k}</math> are independent <math>\chi^{2}_{j}</math> and <math>\chi^{2}_{k}</math> variables, respectively, then <math>W_{j} + W_{k}</math> is <math>\sim \chi^{2}_{j+k}</math>; this result can be extended to any number of independent chi-squared variables. This in turn implies the result the sum of the squares of <math>k</math> independent <math>N(0,1)</math> variables are also <math>\sim \chi^{2}_{k}</math>

Derivation of the <math>\chi^{2}</math> from Gamma functions

Gill discusses the <math>\chi^{2}</math> distribution as a special case of the gamma PDF. That's fine, but there's actually a much more intuitive way of thinking about it, and one that comports more closely with how it is (most commonly) used in statistics. Formally, a variable <math>W</math> that is distributed as <math>\chi^{2}</math> with <math>k</math> degrees of freedom has a density of:

<math>\begin{align} f(w) &=& \frac{1}{2^{k} \Gamma(k)} w^{k} \text{exp} \left[ \frac{-w}{2} \right] \\

              &=& \frac{w^{\frac{k-2}{2}} \exp(\frac{-w}{2})}{2^{\frac{k}{2}} \Gamma(\frac{k}{2})}

\end{align} </math>

where <math>\Gamma(k) = \int_{0}^{\infty} t^{k - 1} \text{exp}(-t) \, dt</math> is the gamma integral (see, e.g., Gill, p.\ 222). As with the normal distribution, the need to write the distribution in this fashion reflects the fact that it has no closed-form solution. The corresponding CDF is

<math> F(w)=\frac{\gamma(k/2,w/2)}{\Gamma(k/2)} </math>

where <math>\Gamma(\cdot)</math> is as before and <math>\gamma(\cdot)</math> is the \texttt{\_Gamma\_function}{lower incomplete gamma function}. We write this\footnote{One also occasionally sees <math>W \sim \chi^{2}(k)</math>, with the degrees of freedom in parentheses.} as <math>W \sim \chi^{2}_{k}</math>, and say ``<math>W</math> is distributed as chi-squared with <math>k</math> degrees of freedom. \\

Additional points needed on the chi-square

Philip Schrodt 07:00, 13 July 2011 (PDT)

  • Probably want to mention the use in contingency tables here, since the connection isn't obvious.
  • Agresti and Finlay state this was introduced by Pearson in 1900, apparently in the context of contingency tables---confirm this, any sort of story here?
  • As df becomes very large, the chi-square approximates the normal; this is a asymptotic distribution and for practical purposes, can be used if df > 50
  • Discuss more about the assumption of statistical independence?
  • Chi-square as the test for comparing whether an observed frequency fits a known distribution

Student's <math>t</math> Distribution

For a variable <math>X</math> which is distributed as <math>t</math> with <math>k</math> degrees of freedom, the PDF function is:

<math> f(x) = \frac{\Gamma(\frac{k+1}{2})} {\sqrt{k\pi}\,\Gamma(\frac{k}{2})} \left(1+\frac{x^2}{k} \right)^{-(\frac{k+1}{2})}\! </math>

where once again <math>\Gamma(\cdot)</math> is the gamma integral. We write <math>X \sim t_{k}</math>, and say ``<math>X</math> is distributed as Student's <math>t</math> with <math>k</math> degrees of freedom. The figure below presents <math>t</math> densities for five different values of <math>k</math>, along with a standard normal density for comparison.


The t-distribution is sometimes known as "Student's t", after a then-anonymous ``student of the statistician Karl Pearson. The story, from Wikipedia,

The t-statistic was introduced in 1908 by William Sealy Gosset, a chemist working for the Guinness brewery in Dublin, Ireland ("Student" was his pen name). Gosset had been hired due to Claude Guinness's innovative policy of recruiting the best graduates from Oxford and Cambridge to apply biochemistry and statistics to Guinness' industrial processes. Gosset devised the t-test as a way to cheaply monitor the quality of stout. He published the test in Biometrika in 1908, but was forced to use a pen name by his employer, who regarded the fact that they were using statistics as a trade secret. In fact, Gosset's identity was unknown to fellow statisticians.

Note a few things about <math>t</math>:

  • The mean/mode/median of a <math>t</math>-distributed variate is zero, and its variance is <math>\frac{k}{k - 2}</math>.
  • <math>t</math> looks like a standard normal distribution (symmetrical, bell-shaped) but has thicker ``tails (read: higher probabilities of draws being relatively far from the mean/mode). However...
  • <math>k</math> gets larger, <math>t</math> converges to a standard normal distribution; at or above <math>k = 30</math> or so, the two are effectively indistinguishable.

The importance of the <math>t</math> distribution lies in its relationship to the normal and chi-square distributions. In particular, if <math>Z \sim N(0,1)</math> and <math>W \sim \chi^{2}_{k}</math>, and <math>Z</math> and <math>W</math> are independent, then

<math>\frac{Z}{\sqrt{W/k}} \sim t_{k} </math>

That is, the ratio of an <math>N(0,1)</math> variable and a (properly transformed) chi-squared variable follows a <math>t</math> distribution, with d.f.\ equal to the number of d.f.\ of the chi-squared variable. Of course, this also means that <math>\frac{Z^{2}}{W/k} \sim t_{k}.</math>

Since we know that <math>Z^{2} \sim \chi^{2}_{1}</math>, this means that another derivation of the <math>t</math> distribution is as a ratio of a <math>\chi^{2}_{1}</math> variate and a <math>\chi^{2}_{k}</math> variate.

Additional points needed on the t distribution

Philip Schrodt 07:00, 13 July 2011 (PDT)

  • May want to note that it is ubiquitous in the inference on regression coefficients
  • Might want to note somewhere---this might go earlier in the discussion of df---that in most social science research (e.g. survey research and time-series cross-sections), the sample sizes are well above the point where the t is asymtotically normal. The t is actually important only in very small samples, though these can be found in situations such as small subsamples in survey research (are Hispanic ferret owners in Wyoming more likely to support the Tea Party?) and situations where the population itself is small (e.g. state membership in the EU, Latin America, or ECOWAS), and experiments with a small number of subjects or cases (this is commonly found in medical research, for example, and this also motivated Gossett's original development of the test, albeit with yeast and hops---we presume---rather than experimental subjects.). In these instances, using the conventional normal approximation to the t---in particular, the rule-of-thumb of looking for standard errors less than twice the size of the coefficient estimate to establish two-tailed 0.05 significance---will be misleading.

The <math>F</math> Distribution

An <math>F</math> distribution is the ratio of two chi-squared variates. If <math>W_{1}</math> and <math>W_{2}</math> are independent and <math>\sim \chi^{2}_{k}</math> and <math>\chi^{2}_{\ell}</math>, respectively, then <math>\frac{W_{1}}{W_{2}} \sim F_{k,\ell} </math>

That is, the ratio of two chi-squared variables is distributed as <math>F</math> with d.f.\ equal to the number of d.f.\ in the numerator and denominator variables, respectively.

Formally, if <math>X</math> is distributed as <math>F</math> with <math>k</math> and <math>\ell</math> degrees of freedom, then the PDF of <math>X</math> is:

<math> f(x) = \frac{\left(\frac{k\,x}{k\,x + \ell}\right)^{k/2} \left(1-\frac{k\,x}{k\,x + \ell}\right)^{\ell/2}}{x\; \mathrm{B}(k/2, \ell/2)} </math>

where <math>\mathrm{B}(\cdot)</math> is the ``beta function. That is, <math>\mathrm{B}(x,y) = \int_0^1t^{x-1}(1-t)^{y-1}\,dt</math>.} We write <math>X \sim F_{k,\ell}</math>, and say ``<math>X</math> is distributed as <math>F</math> with <math>k</math> and <math>\ell</math> degrees of freedom. \\}

The <math>F</math> is a two-parameter distribution, with degrees of freedom parameters (say <math>k</math> and <math>\ell</math>), both of which are limited to the positive integers. An <math>F</math> variate <math>X</math> takes values only on the non-negative real line; it has expected value equal to <math>\text{E}(X) = \frac{\ell}{\ell - 2},</math> which implies that the mean of an <math>F</math>-distributed variable converges on 1.0 as <math>\ell \rightarrow \infty</math>. Likewise, it has variance <math>\text{Var}(X) = \frac{2\,\ell^2\,(k+\ell-2)}{k (\ell-2)^2 (\ell-4)}, </math> which bears no simple relationship to either <math>k</math> or <math>\ell</math>.

The <math>F</math> distribution is (generally) positively skewed. Examples of some <math>F</math> densities with different values of <math>k</math> and <math>\ell</math> are presented in the figure below.


If <math>X \sim F(k, \ell)</math>, then <math>\frac{1}{X} \sim F(\ell, k)</math> (because <math>\frac{1}{X} = \frac{1}{(W_{1} / W_{2})} = \frac{W_{2}}{W_{1}}</math>). In addition, the square of a <math>t</math> distributed variable is <math>\sim F(1,k)</math> (\textit{why}? -- take the formula for <math>t</math>, and square it...)

Additional points needed on the F distribution

Philip Schrodt 10:00, 13 July 2011 (PDT)

  • Discovered by Fisher in 1922, hence "F"
  • Mention how it will be used for <math>R^2</math> and ANOVA <math>F = \frac{MS_{between}}{MS_{within}}</math>
  • Square of a <math>t_k</math> statistic is an <math>F_{1,k}</math> statistic

Summary: Relationships Among Continuous Distributions

The substantive importance of all these distributions will become apparent as we move on to sampling distributions and statistical inference. In the meantime, it is useful to consider the relationship between the four distributions we discussion above



<references group=""></references>

Discussion questions



  • [[Def: ]]
  • [[Def: ]]
  • [[Def: ]]