Difference between revisions of "Statistical distributions"


m (This is a draft taken from my graduate course notes, and is in the process of being edited. PAS EDT)
m (The \chi^{2} Distribution)
Line 135: Line 135:
= The <math>\chi^{2}</math> Distribution =
= The <math>\chi^{2}</math> Distribution =
The <math>\chi^{2}</math>, <math>t</math> and <math>F</math> distributions can be derived from various products of normally-distributed variables.  All three are used extensively in statistical inference and applied statistics, so it's useful to understand them in a bit of depth.  \\
Gill discusses the <math>\chi^{2}</math> distribution as a special case of the gamma PDF.  That's fine, but there's actually a much more intuitive way of thinking about it, and one that comports more closely with how it is (most commonly) used in statistics.  Formally, a variable <math>W</math> that is distributed as <math>\chi^{2}</math> with <math>k</math> degrees of freedom has a density of:
Gill discusses the <math>\chi^{2}</math> distribution as a special case of the gamma PDF.  That's fine, but there's actually a much more intuitive way of thinking about it, and one that comports more closely with how it is (most commonly) used in statistics.  Formally, a variable <math>W</math> that is distributed as <math>\chi^{2}</math> with <math>k</math> degrees of freedom has a density of:

Revision as of 06:18, 13 July 2011



In the previous chapter we discussed probability theory, which we expressed in terms of a variable $X$. We defined $X$ as a set of realizations of some process, which in turn is governed by rules of probability regarding potential outcomes in the sample space.

The variables we were talking about have been what are called random variables, which means that they have a probability distribution. As we noted before, broadly speaking, there are two kinds of random variables: discrete and continuous.

Discrete variables can take on any one of several distinct, mutually-exclusive values.

  • Congressperson's ideology score {0,1,2,3...,100}
  • An individual's political affiliation (Democrat, Republican, Independent}
  • Whether or not a country is a member of European Union (true/false)

A Continuous variable can take on any value in its range.

  • Individual income
  • National population

Need to do

Philip Schrodt 06:57, 13 July 2011 (PDT)

  • Probably need to get most of the probability chapter---which are the moment hasn't been started---written before this one. In particular, will the pdf and cdf be defined there or here?
  • Add some of the discrete distributions, particularly the binomial
  • Add the uniform?
  • Do we add---or link to on another page---the derivation of the mean and standard errors for these: that code is available in CCL on an assortment of places on the web

The Normal Distribution

We are all used to seeing normal distributions described, and to hearing that something is "normally distributed." We know that a normal distribution is "bell-shaped," and symmetrical, and probably that it has some mean and some standard deviation.

Formally, if <math>X</math> is a normally distributed variate with mean <math>\mu</math> and variance <math>\sigma^{2}</math>, then:

<math>f(x) = \frac{1}{\sigma \sqrt{2\pi}} \text{exp} \left( - \frac{(x - \mu)^{2}}{2 \sigma^{2}} \right)</math>.

We denote this <math>X \sim N(\mu,\sigma^{2})</math>, and say ``<math>X</math> is distributed normally with mean mu and variance sigma squared. The symbol <math>\phi</math> is often used as a shorthand to represent the normal density in \eqref{normalden}:

<math>X \sim \phi_{\mu, \sigma^{2}}</math>.

The corresponding normal CDF -- which is the probability of a normal random variate taking on a value less than or equal to some specified number -- is (as always) the indefinite integral of \eqref{normalden}. This has no simple closed-form solution, so we typically just write:

<math>F(x) \equiv \Phi_{\mu, \sigma^{2}}(x) = \int \phi_{\mu, \sigma^{2}} f(x) d x.</math>

Here are a bunch of normal curves


Normal cumulative distribution functions

Bases for the Normal Distribution

The most common justification for the normal distribution has its roots in the 'central limit theorem'. Consider <math>i = {1,2,...N}</math> independent, real-valued random variates $<math>X_{i}</math>$, each with finite mean $<math>\mu_{i}</math>$ and variance <math>\sigma^{2}_{i} > 0</math>. If we consider a new variable $<math>X</math>$ defined as the sum of these variables:

<math>X = \sum_{i=1}^{N} X_{i}</math>

then we know that

<math> \text{E}(X) = \sum_{i=1}^{N} \mu_{i} </math>


<math> \text{Var}(X) = \sum_{i=1}^{N} \sigma^{2}_{i} </math>

The central limit theorem states that:

<math> \underset{N \rightarrow \infty}{\lim} X = \underset{N \rightarrow \infty}{\lim} \sum_{i=1}^{N} X_{i} \overset{D}{\rightarrow} N(\cdot) </math>

where the notation <math>\overset{D}{\rightarrow}</math> indicates convergence in distribution. That is, as <math>N</math> gets sufficiently large, the distribution of the sum of <math>N</math> independent random variates with finite mean and variance will converge to a normal distribution. As such, we often think of a normal distribution as being appropriate when the observed variable <math>X</math> can take on a range of continuous values, and when the observed value of <math>X</math> can be thought of as the product of a large number of relatively small, independent ``shocks or perturbations.

Properties of the Normal Distribution

  • A normal variate <math>X</math> has support in <math>\mathfrak{R}</math>.
  • The normal is a two-parameter distribution, where <math>\mu \in (-\infty, \infty)</math> and <math>\sigma^{2} \in (0, \infty)</math>.
  • The normal distribution is always symmetrical (<math>M_{3} = 0</math>) and mesokurtic.
  • item The normal distribution is preserved under a linear transformation. That is, if <math>X \sim N(\mu,\sigma^{2})</math>, then <math>aX + b \sim N(a\mu + b, a^{2} \sigma^{2})</math>. (Why? Recall our earlier results on <math>\mu</math> and <math>\sigma^{2}</math>).

The Standard Normal Distribution

One linear transformation is especially useful:

<math> \begin{align}

b & = \frac{-\mu}{\sigma} \\
a & = \frac{1}{\sigma}

\end{align} </math>.

This yields:

<math> \begin{align}

ax + b  & \sim  N(a\mu+b, a^{2} \sigma^{2}) \\ 
        &  \sim N(0,1) 

\end{align} </math>

This is the standard normal density function. We often denote this <math>\phi(\cdot)</math>, and say that "X is distributed as standard normal." We can also get this by transforming ("standardizing") the normal variate <math>X</math>...

  • If <math>X \sim N(\mu,\sigma^{2})</math>, then <math>Z = \frac{(x - \mu)}{\sigma} \sim N(0,1)</math>.
  • The density function then reduces to:

<math> f(z) = \equiv \phi(z) = \frac{1}{\sqrt{2\pi}} \text{exp} \left[ - \frac{(z)^{2}}{2} \right] </math>

Similarly, we often write the CDF for the standard normal as <math>\Phi(\cdot)</math>.

Why do we care about the normal distribution?

The normal distribution's importance lies in its relationship to the central limit theorem. As we'll discuss at more length later, the central limit theorem means that as one's sample size increases, the distribution of sample means (or other estimates) approaches a normal distribution.

Additional points needed on the normal

Philip Schrodt 07:00, 13 July 2011 (PDT)

  • More extended discussion of the CLT, and a note that if we are dealing with a data generating process where the "error" is the average (or cumulative) effect of a large number of random variables with a variety of distributions, the CLT tells us that the net effect will be normally distributed. This, in turn, explains why linear models that assume Normally distributed error---regression and ANOVA---have proven to be so robust in practice
  • Link to a number of examples of normally distributed data...should be easy to find these on the web. E.g. the classical height. Maybe SAT scores, though these are artificially normal
  • ref to the wikipedia article; there is also a nice graphic to snag from there---introductory sidebar---which shows the standard normal
  • sidebar on the log-normal?
  • something about the bivariate normal and some nice graphics of this?
  • sidebar on the issue of fat tails and how these destroyed the economy in 2007?---there is a fairly readable Wired article on this:

The <math>\chi^{2}</math> Distribution

Gill discusses the <math>\chi^{2}</math> distribution as a special case of the gamma PDF. That's fine, but there's actually a much more intuitive way of thinking about it, and one that comports more closely with how it is (most commonly) used in statistics. Formally, a variable <math>W</math> that is distributed as <math>\chi^{2}</math> with <math>k</math> degrees of freedom has a density of:

<math>\begin{align} f(w) &=& \frac{1}{2^{k} \Gamma(k)} w^{k} \text{exp} \left[ \frac{-w}{2} \right] \\

              &=& \frac{w^{\frac{k-2}{2}} \exp(\frac{-w}{2})}{2^{\frac{k}{2}} \Gamma(\frac{k}{2})}

\end{align} </math>

where <math>\Gamma(k) = \int_{0}^{\infty} t^{k - 1} \text{exp}(-t) \, dt</math> is the gamma integral (see, e.g., Gill, p.\ 222). As with the normal distribution, the need to write the distribution in this fashion reflects the fact that it has no closed-form solution. The corresponding CDF is

<math> F(w)=\frac{\gamma(k/2,w/2)}{\Gamma(k/2)} </math>

where <math>\Gamma(\cdot)</math> is as before and <math>\gamma(\cdot)</math> is the \texttt{\_Gamma\_function}{lower incomplete gamma function}. We write this\footnote{One also occasionally sees <math>W \sim \chi^{2}(k)</math>, with the degrees of freedom in parentheses.} as <math>W \sim \chi^{2}_{k}</math>, and say ``<math>W</math> is distributed as chi-squared with <math>k</math> degrees of freedom. \\

The chi-square distribution is a one-parameter distribution defined only on the nonnegative real line, <math>W \in [0, \infty)</math>. It is positively skewed, with <math>\text{E}(W) = k </math>

and <math>\text{Var}(W) = 2k. </math>

Figure \ref{ChiSquares} presents five <math>\chi^{2}</math> densities with different values of <math>k</math>.


Characteristics of the <math>\chi^{2}</math> Distribution

More importantly, one needs to remember two key things about the chi-square distribution:

  • If <math>Z \sim N(0,1)</math>, then <math>Z^{2} \sim \chi^{2}_{1}</math>. That is, \emph{the square of a <math>N(0,1)</math> variable is chi-squared with one degree of freedom}.
  • If <math>W_{j}</math> and <math>W_{k}</math> are independent <math>\chi^{2}_{j}</math> and <math>\chi^{2}_{k}</math> variables, respectively, then <math>W_{j} + W_{k}</math> is <math>\sim \chi^{2}_{j+k}</math>; this result can be extended to any number of independent chi-squared variables.

The first of these is key, since it points out that the square of a standard normal variate is a one-degree-of-freedom chi-square variable. This explains why (e.g.) a chi-squared variate only has support on the nonnegative real numbers. The second point is also tremendously useful to know, in that it has a number of valuable corollaries. For example, it implies that

  • if <math>W_{1},W_{2},...W_{k}</math> are all independent <math>\chi^{2}_{1}</math> variables, then <math>\sum_{i=1}^{k}W_{i} \sim \chi^{2}_{k}</math>. (The sum of <math>k</math> independent chi-squared variables is chi-squared with <math>k</math> degrees of freedom).
  • By extension, the sum of the squares of <math>k</math> independent <math>N(0,1)</math> variables are also <math>\sim \chi^{2}_{k}</math>.

All this means that, if we know a variable to be normally distributed, we can consider its squared, standardized values to be <math>\chi^{2}_{1}</math>, and the sums of <math>k</math> of such variables to be <math>\chi^{2}_{1}</math>.

Additional points needed on the chi-square

Philip Schrodt 07:00, 13 July 2011 (PDT)

Student's <math>t</math> Distribution

For a variable <math>X</math> which is distributed as <math>t</math> with <math>k</math> degrees of freedom, the PDF function is:

<math> f(x) = \frac{\Gamma(\frac{k+1}{2})} {\sqrt{k\pi}\,\Gamma(\frac{k}{2})} \left(1+\frac{x^2}{k} \right)^{-(\frac{k+1}{2})}\! </math>

where once again <math>\Gamma(\cdot)</math> is the gamma integral. We write <math>X \sim t_{k}</math>, and say ``<math>X</math> is distributed as Student's <math>t</math> with <math>k</math> degrees of freedom. The CDF is complicated, so I won't go into it here; Figure \ref{Ts} presents <math>t</math> densities for five different values of <math>k</math>, along with a standard normal density for comparison.


The t-distribution is sometimes known as "Student's t", after a then-anonymous ``student of the statistician Karl Pearson. The story, from Wikipedia,

The t-statistic was introduced in 1908 by William Sealy Gosset, a chemist working for the Guinness brewery in Dublin, Ireland ("Student" was his pen name). Gosset had been hired due to Claude Guinness's innovative policy of recruiting the best graduates from Oxford and Cambridge to apply biochemistry and statistics to Guinness' industrial processes. Gosset devised the t-test as a way to cheaply monitor the quality of stout. He published the test in Biometrika in 1908, but was forced to use a pen name by his employer, who regarded the fact that they were using statistics as a trade secret. In fact, Gosset's identity was unknown to fellow statisticians.

Note a few things about <math>t</math>:

  • The mean/mode/median of a <math>t</math>-distributed variate is zero, and its variance is <math>\frac{k}{k - 2}</math>.
  • <math>t</math> looks like a standard normal distribution (symmetrical, bell-shaped) but has thicker ``tails (read: higher probabilities of draws being relatively far from the mean/mode). However...
  • <math>k</math> gets larger, <math>t</math> converges to a standard normal distribution; at or above <math>k = 30</math> or so, the two are effectively indistinguishable.

The importance of the <math>t</math> distribution lies in its relationship to the normal and chi-square distributions. In particular, if <math>Z \sim N(0,1)</math> and <math>W \sim \chi^{2}_{k}</math>, and <math>Z</math> and <math>W</math> are independent, then

<math>\frac{Z}{\sqrt{W/k}} \sim t_{k} </math>

That is, the ratio of an <math>N(0,1)</math> variable and a (properly transformed) chi-squared variable follows a <math>t</math> distribution, with d.f.\ equal to the number of d.f.\ of the chi-squared variable. Of course, this also means that <math>\frac{Z^{2}}{W/k} \sim t_{k}.</math>

Since we know that <math>Z^{2} \sim \chi^{2}_{1}</math>, this means that another derivation of the <math>t</math> distribution is as a ratio of a <math>\chi^{2}_{1}</math> variate and a <math>\chi^{2}_{k}</math> variate. As we'll see in a week or so, that turns out to be quite important, and useful.

Additional points needed on the t distribution

Philip Schrodt 07:00, 13 July 2011 (PDT)

The <math>F</math> Distribution

An <math>F</math> distribution is best understood as the ratio of two chi-squared variates. Formally, if <math>X</math> is distributed as <math>F</math> with <math>k</math> and <math>\ell</math> degrees of freedom, then the PDF of <math>X</math> is:

<math> f(x) = \frac{\left(\frac{k\,x}{k\,x + \ell}\right)^{k/2} \left(1-\frac{k\,x}{k\,x + \ell}\right)^{\ell/2}}{x\; \mathrm{B}(k/2, \ell/2)} </math>

%where <math>\mathrm{B}(\cdot)</math> is the ``beta function.\footnote{That is, <math>\mathrm{B}(x,y) = \int_0^1t^{x-1}(1-t)^{y-1}\,dt</math>.} The corresponding CDF is (once again) complicated, so we'll skip it. We write <math>X \sim F_{k,\ell}</math>, and say ``<math>X</math> is distributed as <math>F</math> with <math>k</math> and <math>\ell</math> degrees of freedom. \\}

The <math>F</math> is a two-parameter distribution, with degrees of freedom parameters (say <math>k</math> and <math>\ell</math>), both of which are limited to the positive integers. An <math>F</math> variate <math>X</math> has support on the non-negative real line; it has expected value equal to <math>\text{E}(X) = \frac{\ell}{\ell - 2}, </math>

which implies that the mean of an <math>F</math>-distributed variable converges on 1.0 as <math>\ell \rightarrow \infty</math>. Likewise, it has variance <math>\text{Var}(X) = \frac{2\,\ell^2\,(k+\ell-2)}{k (\ell-2)^2 (\ell-4)}, </math>

which bears no simple relationship to either <math>k</math> or <math>\ell</math>. It is (generally) positively skewed. Examples of some <math>F</math> densities with different values of <math>k</math> and <math>\ell</math> are presented in Figure \ref{Fs}.


As I noted a minute ago, if <math>W_{1}</math> and <math>W_{2}</math> are independent and <math>\sim \chi^{2}_{k}</math> and <math>\chi^{2}_{\ell}</math>, respectively, then <math>\frac{W_{1}}{W_{2}} \sim F_{k,\ell} </math>

That is, the ratio of two chi-squared variables is distributed as <math>F</math> with d.f.\ equal to the number of d.f.\ in the numerator and denominator variables, respectively. This implies (at least) a couple of interesting things:

  • If <math>X \sim F(k, \ell)</math>, then <math>\frac{1}{X} \sim F(\ell, k)</math> (because <math>\frac{1}{X} = \frac{1}{(W_{1} / W_{2})} = \frac{W_{2}}{W_{1}}</math>).
  • The square of a <math>t</math> distributed variable is <math>\sim F(1,k)</math> (\textit{why}? -- take the formula for <math>t</math>, and square it...)

Additional points needed on the F distribution

Philip Schrodt 07:00, 13 July 2011 (PDT)

Summary: Relationships Among Continuous Distributions

The substantive importance of all these distributions will become apparent as we move on to sampling distributions and statistical inference. In the meantime, it is useful to consider the relationship between the four distributions we discussion above



<references group=""></references>

Discussion questions



  • [[Def: ]]
  • [[Def: ]]
  • [[Def: ]]