==========================================
 Probability Space and Probability Axioms
==========================================




Random Experiments and Probability Spaces
=========================================

To describe a random experiment in mathematical terms we need what is
called a probability space. Such a space consists of three things:

#. The set of all possible outcomes of the random experiment. We will
   denote this set as $U$ for **universe**.

#. A collection of subsets of $U$. A subset $A$ if $U$ (i.e. $A\subset
   U$) corresponds with an **event**. For example when throwing a die
   a possible event is to throw a 6. Another possibiliy is the event
   corresponding with throwing an odd number of points.

#. A probability measure $\P$ that maps each subset $A\subset U$
   (event) onto a positive scalar $\P(A)$



Probability Axioms
==================

Let $A$ and $B$ both be subsets of the probability space (the
universe) $U$, i.e. $A$ and $B$ are events. Surprisingly maybe, the
whole theory of probability is based upon just three axioms:

.. proof:axiom:: Probability Axioms

   .. math::
      (A_1)&\quad \P(A)\geq 0\\
      (A_2)&\quad \P(U)=1\\
      (A_3)&\quad A\cap B=\emptyset \Rightarrow \P(A\cup B) = \P(A) + \P(B)

The first axiom simply states that probabilities are non negative
(real) numbers.

The second axioma states that the probability for the universe $U$
equals 1. This is intuitively clear as the universe contains
everything that could ever happen.

The third axiom is the only axiom that relates probabilities of
several events: if event $A$ and event $B$ are mutually exclusive
(i.e. you can't have $A$ and $B$ being the simulatanuous outcome of
one random experiment, e.g. throwing with a die and having events "6"
and "odd") then the probability of either event $A$ or event $B$
occuring, i.e. $A\cup B$, equals the sum of $\P(A)$ and $\P(B)$.


Venn Diagrams
=============

To gain insight and some intuition of probability space it is useful
to depict that space as a two dimensional subset of the plane (the
**Venn diagrams**  of sets).

.. figure:: /figures/venn_disjunct.png
   :width: 30%
   :align: center

   **Disjunct sets.** The sets $A$ and $B$ have nothing in common,
   i.e. $A\cap B=\emptyset$.


Consider the Venn diagram corresponding with the third axiom. We have
the universe $U$ and the two events $A$ and $B$, both subset of
$U$. The assumption that $A\cap B =\emptyset$ is visualised as
the empty intersection of the two sets.

You may take the visual interpretation somewhat further and assume the
relative area of set $A$ with respect to the area of set $U$
corresponds with the probability $\P(A)$:

.. math::
   \P(A) = \frac{\text{area}(A)}{\text{area}(U)}

Please note that this is NOT the definition of $\P(A)$ ! It is stating
the rule to interpret the Venn diagrams.


Simple Probability Theorems
===========================

The definitions and axioms in a mathematical theory are the starting
point, but the interesting parts are the theorems, the things that can
be proven to be true given the definitions and the axioms. Probability
theory is no exception.

Let us start with a simple definition: the complement. Let $A$ be an
event, i.e. subset of $U$. Then we define the **complement** $\neg A$
as:

.. math::
   \neg A = U \setminus A

where $\setminus$ denotes the 'set minus', i.e. everything in the
universe that is not in $A$. This is just a definition. A theorem
however is the statement:


.. proof:theorem::

   .. math::
      \P(\neg A) = 1 - \P(A)

.. proof:proof::

   The proof obviously needs the third axiom as we have to relate the
   probabilities of two events: $A$ and $\neg A$. Observe that:

   .. math::
      U = A \cup \neg A

   By definition we have that $A$ and $\neg A$ are **disjunct**,
   i.e. $A\cap \neg A=\emptyset$, and thus we can utilize the third
   axiom:

   .. math::
      \P(U) = \P(A) + \P(\neg A)

   Using the second axiom and rewriting the equation we get:

   .. math::
      \P(\neg A) = 1 - \P(A)

   and that completes the proof.

.. figure:: /figures/venn_AcapB.png
   :width: 30%
   :align: center

   **Intersection of sets.** The intersection $A\cap B$ is sketched as
   the hatched area.


Now consider the situation where events $A$ and $B$ are not
disjunct. Then we cannot apply the third axiom directly. Look at a
Venn diagram of two subsets $A$ and $B$ or $U$ with non empty
intersection. From such a figure it is intuively clear that


.. proof:theorem::

   .. math::
      \P(A\cup B) = \P(A) + \P(B) - \P(A\cap B).

.. proof:proof::

   The proof is simple as well. We leave that to the reader in an
   exercise.