2. Probability and Statistics

What is the probability of throwing a 6 with a fair die? Almost everyone immediately will answer: \(1/6\) -th. But what do we mean by that? Two obvious lines of reasoning to arrive at that answer are:

• If we throw a die there are six possible outcomes and because it is a fair die every outcome is equally probable, the probabilities of each possible outcome should add up to one, and so the probability of throwing a 6 is equal to \(1/6\) -th.

• Let’s repeat the experiment a large number of times, say \(N\). After throwing the die \(N\) times we estimate the probability as:

  \[\text{Probability(throwing 6)} = \frac{\text{#\{throwing 6\}}}{N}\]

  This estimate will approach the true probability if we let \(N\rightarrow\infty\).

This line of reasoning is part of what is called the frequentist approach to probability. We will follow this route quite often in this course.

But now consider the question: what is the probability that it will rain tomorrow? Neither of the two lines of reasoning above can help us here? Tomorrow it will either rain or not but these possible outcomes are clearly not necessarily equally probable. And if we repeat the experiment \(N\) times (say we take \(N=10\times365\), then we have data over 10 years) and use the frequentist approach we undoubtly end up with the probability that it will rain on any randomly given day in the Netherlands. Certainly not the probability that it will rain tomorrow. Assigning a probability of 70% for rain tomorrow quantifies our belief that it will rain tomorrow. A belief that is based on data and models.

Fortunately the interpretation of probability (being a frequentist view or a belief based view) is of lesser importance. The classical mathematical rules for probability and statistics that we will look at in this course are largely independent of the interpretation. Loosely speaking we might say that probability is the mathematical language of choice when dealing with uncertainty.

Probabilty theory deals with random experiments and random processes. Experiments and processes that are not deterministic, it is simply not possible to know beforehand exactly what will be the result of such an experiment or process. In probability theory we assume that for each of the possible outcomes of a random experiment the corresponding probabilities are well defined.

Fig. 2.1 The Dutch term “steekproef” for a statistical sample, comes from the way Dutch cheese is tested on the traditional cheesemarket in Alkmaar. A cylindrical sample out of a cheese is tested to assess the quality of several cheeses.

With statistics we enter the realm of everyday life. We may assume that there exists a function that assigns a probability to each possible outcome of a random experiment, but alas all we have are observations (a sample or ‘steekproef’ in Dutch) of the random experiment. Statistics then deals (among others) with the question what can be known about the underlying random experiment using only the observations in the sample? Because the sample consists of random numbers the conclusions from statistics about the random experiment are in a sense random as well. We can never be completely sure about our (numerical) conclusions. A lot a statistics deals with this important question: can we quantize the probability that we arrive at the right (or wrong) conclusions?

• 2.1. Probability Space and Probability Axioms
  • 2.1.1. Random Experiments and Probability Spaces
  • 2.1.2. Probability Axioms
  • 2.1.3. Venn Diagrams
  • 2.1.4. Simple Probability Theorems
• 2.2. Conditional Probabilities
• 2.3. Probability Trees
• 2.4. Independent Events
• 2.5. Random Variables
• 2.6. Probability Distributions
  • 2.6.1. Discrete Distributions
    • 2.6.1.1. Bernoulli Distribution
    • 2.6.1.2. Binomial Distribution
    • 2.6.1.3. Uniform Distribution
  • 2.6.2. Continuous Distributions
    • 2.6.2.1. Uniform Distribution
    • 2.6.2.2. Normal Distribution
• 2.7. Joint Distributions
  • 2.7.1. Two Discrete Distributions
  • 2.7.2. Two Continuous Distributions
  • 2.7.3. Multiple Random Variables
• 2.8. Calculations with Random Variables
  • 2.8.1. Scaled and Shifted Distributions
  • 2.8.2. The Sum of Random Variables
• 2.9. Descriptive Statistics
  • 2.9.1. Histograms
  • 2.9.2. Statistics
• 2.10. Estimators for Distribution Parameters
  • 2.10.1. Maximum Likelihood Estimator
  • 2.10.2. Bernoulli Distribution
  • 2.10.3. Categorical Distribution
  • 2.10.4. Normal Distribution
• 2.11. Random Vectors
  • 2.11.1. Expectation and (Co)variance
    • 2.11.1.1. The Expectation of Random Vectors
    • 2.11.1.2. The (Co)Variance of Random Vectors
    • 2.11.1.3. Directional variance
    • 2.11.1.4. Correlation and Dependency
    • 2.11.1.5. Estimation of \(E(\v X)\) and \(\Cov(\v X)\)
  • 2.11.2. Normal Distributed Random Vectors
    • 2.11.2.1. Definition
    • 2.11.2.2. Geometry
• 2.12. Exercises