===========================
 Probability Distributions
===========================

There are a lot of probability mass functions and probability density
functions for which an analytical expression is known. These functions
are parameterized in the sense that only a few parameters define the
value for every value $x$.

In this section we discuss some of the most often used functions. In a
later section we describe one method to estimate the parameters given
a sample from a distribution.



Discrete Distributions
----------------------

A discrete distribution is completely characterized with its
probability mass function $p_X$. Remember that for a probability mass
function we should have that the sum over all possible outcomes $x$
equals 1 and that all probabilities are greater than or equal to
zero. Given that function we will calculate (or only give the result)
both the expectation and the variance.



Bernoulli Distribution
.......................

.. figure:: /figures/headstails.jpg
   :figwidth: 40%
   :align: right

   **Heads or Tails.** Grammarist explains. “Heads refers to the side
   of the coin with a person’s head on it. Tails refers to the
   opposite side, not because there is a tail on it, but because it is
   the opposite of heads.”

The simplest of all probabilistic experiments perhaps is tossing a
coin. The outcome is either heads or tails, or 0 or 1. Such an
experiment is called a Bernoully experiment and the corresponding
distribution is called the **Bernoulli distribution**.

The Bernoulli distribution has parameter $p$. In case the random
variable $X$ is Bernoulli distributed we write
$X\sim\Bernoulli(p)$.

The probability mass function is:

.. math::
   p_X(x) = \begin{cases}
   1-p &: x=0\\
   p &: x=1\\
   0 &: \text{elsewhere}
   \end{cases}

The expectation equals:

.. math::
   \E(X) &= \sum_{x=-\infty}^{\infty} x\,p_X(x)\\
         &= 0\times (1-p) + 1\times p\\
         &= p

and its variance:

.. math::
   \Var(X) = p(1-p)



Binomial Distribution
.....................

.. figure:: /figures/galton.gif
   :figwidth: 50%
   :align: right

   **Galton Board.** A simulation of a Galton board. The distribution
   of the ball over the bins follows binomial distribution. Open the
   animated gif image in a separate window to clearly see the pins
   that force a falling ball to go either left or right (the
   Bernouilli experiment). Google for Galton board video and you can
   see a physical Galton board in action.

Consider a Bernoulli distributed random variable
$Y\sim\Bernoulli(p)$, and let us repeat the experiment $n$
times. What then is the probabilty of $k$ successes. A success is
defined as the outcome $Y=1$ for the Bernoulli experiment.

So we define a new random variable $X$ that is the sum of $n$ outcomes
of repeated independent and identicallu distributed (iid) Bernoulli
experiments.

The outcomes of $X$ run from $0$ to $n$ and the probability of finding
$k$ successes is given as:

.. math::
   p_X(k) = P(X=k) =  {n \choose k}\,p^k\,(1-p)^{n-k}

This is called the **Binomial Distribution**. For a random variable
$X$ that has a binomial distribution we write $X\sim\Bin(n,p)$.

The expectation is:

.. math::
   \E(X) = n\,p

and the variance:

.. math::
   \Var(X) = n\,p\,(1-p)

.. exec_python:: rvPMF rvDiscrete
   :linenumbers:
   :code: shutter
   :Code_label: Show code for figure
   :results: hide

   import numpy as np
   import matplotlib.pyplot as plt

   from scipy.stats import binom

   n = 20
   for p, c in zip([0.05, 0.4, 0.8], ['r', 'g', 'b']):
       plt.stem(np.arange(0, n+1), binom.pmf(np.arange(0, n+1), n, p),
                linefmt=c, markerfmt=c+'o',
                label=f'p={p}')

       plt.legend()
   plt.xticks(np.arange(0, n+1))
   plt.savefig('source/figures/binomialdistribution.png')


.. figure:: /figures/binomialdistribution.png
   :align: center
   :width: 60%

   **Binomial Distribution.** Shown are the binomial probability mass
   functions for $n=20$ and $p\in\{0.05, 0.4, 0.8\}$.

Uniform Distribution
....................

.. figure:: /figures/dice12.jpg
   :figwidth: 50%
   :align: right

   **Twelve sided die.** Throwing this die result in a number in the
   range from 1 to 12 to show on top with equal probability (in case
   of a fair die).

The **discrete uniform distribution** is used in case all possibe
outcomes of a random experiment are equally probable. Let
$X\sim\Uniform(a,b)$, with $a<b$ and $a,b\in\setZ$, then

.. math::
   p_X(x) = \begin{cases}
   \frac{1}{b-a+1} &: a\leq x \leq b\\
   0 &: \text{elsewhere}
   \end{cases}

Its expectation is:

.. math::
   \E(X) = \frac{a+b}{2}

and its variance

.. math::
   \Var(X) = \frac{(b-a+1)^2-1}{12}




Continuous Distributions
------------------------

Uniform Distribution
....................

The **continuous uniform distribution** characterizes a random
experiment in which each real valued outcome in the interval
$[a,b]\subset\setR$ is equally probable. The probability density
function of $X\sim\Uniform(a,b)$ is given by

.. math::
   f_X(x) = \begin{cases}
   \frac{1}{b-a} &: a \leq x \leq b\\
   0 &: \text{elsewhere}
   \end{cases}


The expectation of the continuous uniform distribution is

.. math::
   \E(X) = \frac{a+b}{2}

and its variance

.. math::
   \Var(X) = \frac{(b-a)^2}{12}



Normal Distribution
...................

The **normal distribution** is undoubtly the most often used
distribution in probability and machine learning. For good reasons: a
lot of natural phenomena of random character turn out to be normally
distributed (at least in good approximation). Furthermore the normal
distribution is a nice one to work with from a mathematical point of
view.

Let $X\sim\Normal(\mu,\sigma^2)$ then the probability density function
is:

.. math::
   f_X(x) = \frac{1}{\sigma\sqrt{2 \pi}} e^{-\frac{(x-\mu)^2}{2\sigma^2}}


where the parameters $\mu$ and $\sigma^2$ are called the mean and
variance for obvious reasons:

.. math::
   &\E(X) = \mu\\
   &\Var(X) = \sigma^2

There is no analytical expression for the cumulative distribution
function $F_X$ for $X\sim N(0,1)$ (nor for any normal
distribution). We Instead we write:

.. math::
   F_X(x) = \frac{1}{2}\left(1+\text{erf}\frac{x}{\sqrt{2}}\right)

where $\text{erf}$ is the error function

.. math::
   \text{erf}(x) = \frac{2}{\sqrt{\pi}} \int_0^x e^{-t^2} dt

The error function is available in Numpy as :code:`scipy.special.erf`. Given
$F_X(x)$ we can easily calculate the probability for any outcome in
the interval from $a$ to $b%:

.. math::
   \P(a\leq x \leq b) = F_X(b) - F_X(a)

Famously we have:

.. math::
   \P(\mu - \sigma\leq &x \leq \mu+\sigma) = 0.683\\
   \P(\mu - 2\sigma\leq &x \leq \mu+2\sigma) = 0.954\\
   \P(\mu - 3\sigma\leq &x \leq \mu+3\sigma) = 0.997

which are illustrated in the figure below:

.. exec_python:: rvNormal rvContinuous
   :linenumbers:
   :code: shutter
   :Code_label: Show code for figure
   :results: hide

   import numpy as np
   import matplotlib.pyplot as plt
   from scipy.special import erf

   def Phi(x):
       return 0.5 * (1 + erf(x / np.sqrt(2)))

   def phi(x):
       return 1/np.sqrt(2*np.pi) * np.exp(-x**2/2)

   fig, axs = plt.subplots(3)
   x = np.linspace(-4, 4, 1000)

   for i, s in enumerate([1,2,3]):
      axs[i].plot(x, phi(x))
      xi = x[np.logical_and(x>-s, x<s)]
      axs[i].fill_between(xi, 0, phi(xi))
      axs[i].text(2, 0.3, "P = {P:5.3f}".format(P=Phi(s)-Phi(-s)))
      plt.axhline(0)

   plt.savefig('source/figures/normalintervals.png')


.. figure:: /figures/normalintervals.png
   :align: center
   :width: 60%

   **Normal Distribution.** Shown are the intervals from $\mu-k\sigma$
   to $\mu + k\sigma$ and their probability $\P(\mu-k\sigma\leq x
   \leq \mu+k\sigma)$ for $k=1,2,3$.