Probability Distributions
=========================

Discrete Distributions
----------------------

Bernoulli Distribution
.......................

The simplest of all probabilistic experiments perhaps is tossing a
coin. The outcome is either heads or tails, or 0 or 1, or.. Such an
experiment is called a Bernoully experiment and the corresponding
distribution is called the **Bernoulli distribution**.

The Bernoulli distribution has parameter $p$. In case the random
variable $X$ is Bernoulli distributed we write
$X\sim\Bernoulli(p)$.

The probability mass function is:

.. math::
   p_X(x) = \begin{cases}
   1-p &: x=0\\
   p &: x=1\\
   0 &: \text{elsewhere}
   \end{cases}

The expectation equals:

.. math::
   \E(X) &= \sum_{x=-\infty}^{\infty} x\,p_X(x)\\
         &= p

and its variance:

.. math::
   \Var(X) = p(1-p)



Binomial Distribution
.....................

Consider a Bernoulli distributed random variable
$Y\sim\Bernoulli(p)$, and let us repeat the experiment $n$
times. What then is the probabilty of $k$ successes. A success is
defined as the outcome $Y=1$ for the Bernoulli experiment.

So we define a new random variable $X$ that is the sum of $n$ outcomes
of repeated independent and identicallu distributed (iid) Bernoulli
experiments.

The outcomes of $X$ run from $0$ to $n$ and the probability of finding
$k$ successes is given as:

.. math::
   p_X(x) = P(X=k) =  {n \choose k}\,p^k\,(1-p)^{n-k}

This is called the **Binomial Distribution**. For a random variable
$X$ that has a binomial distribution we write $X\sim\Bin(n,p)$.

The expectation is:

.. math::
   \E(X) = n\,p

and the variance:

.. math::
   \Var(X) = n\,p\,(1-p)

   
Uniform Distribution
....................

The **discrete uniform distribution** is used in case all possibe
outcomes of a random experiment are equally probable. Let
$X\sim\Uniform(a,b)$, with $a<b$ and $a,b\in\setZ$, then

.. math::
   p_X(x) = \begin{cases}
   \frac{1}{b-a+1} &: a\leq x \leq b\\
   0 &: \text{elsewhere}
   \end{cases}

Its expectation is:

.. math::
   \E(X) = \frac{a+b}{2}

and its variance

.. math::
   \Var(X) = \frac{(b-a+1)^2-1}{12}

   


Continuous Distributions
------------------------

Uniform Distribution
....................

The **continuous uniform distribution** characterizes a random
experiment in which each real valued outcome in the interval
$[a,b]\subset\setR$ is equally probable. The probability density
function of $X\sim\Uniform(a,b)$ is given by

.. math::
   f_X(x) = \begin{cases}
   \frac{1}{b-a} &: a \leq x \leq b\\
   0 &: \text{elsewhere}
   \end{cases}


   

Normal Distribution
...................

The **normal distribution** is undoubtly the most often used
distribution in probability and machine learning. For good reasons: a
lot of natural phenomena of random character turn out to be normally
distributed (at least in good approximation). Furthermore the normal
distribution is a nice one to work with from a mathematical point of
view.

Let $X\sim\Normal(\mu,\sigma^2)$ then the probability density function
is:

.. math::
   f_X(x) = \frac{1}{\sigma\sqrt{2 \pi}} e^{-\frac{(x-\mu)^2}{2\sigma^2}}

   
where the parameters $\mu$ and $\sigma^2$ are called the mean and
variance for good reason:

.. math::
   &\E(X) = \mu\\
   &\Var(X) = \sigma^2