introduction to probability

1 - What is Probability?

1.1 - Experiments

An experiment is a procedure, subject to uncertainty, which necessarily produces an outcome.

A collection of outcomes is called an event.
The collection of all outcomes is called the sample space.
The collection of all events to be considered is the event space.

An event is said to occur if it contains the observed outcome.
Probability concerns events and how likely they are to occur.

1.2 - Probability Axioms

Probability is axiomatized by the probability axioms:

the probability of an event is a non-negative real number
the probability of the entire sample space is 1
any mutually exclusive events have probability which is additive

1.3 - Probability as a Measure

A model is a mathematical tool which systematizes a set of axioms to something which can be manipulated. See probability space for the rigorous model in the most technical language.

1.3.1 - Definitions

Here, we define

the sample space $Ω$ is the set of all outcomes,
an event $E \subseteq Ω$ is a subset of outcomes,
the set of all events $F = {E : E \subseteq Ω}$ is the event space
Then the probability measure $P : F \to [0, 1]$ is a measure (a type of function) which takes an event $E \in F$ and assigns it a real number $P (E) \in [0, 1]$ .

1.3.2 - Interpreting the Axioms

Now, the probability axioms can be rewritten

$P (E) \in [0, 1]$ for all events $E$
$P (Ω) = 1$
$P (⋃ E_{i}) = \sum P (E_{i})$ for mutually exclusive events $E_{i}$

1.3.3 - Corollaries to the Axioms

$P (\emptyset) = 0$
$E \subseteq F ⟹ P (E) \leq P (F)$

1.4 - Competing Interpretations

Plainly: For each event $A$ , there is some associated probability $P (A)$ .

Under the frequentist interpretation of probability, $P (A)$ is the frequency with which $A$ would occur under a theoretically infinite number of repetitions of the experiment (i.e. the law of large numbers)

Under the statistical interpretation of probability, $P (A)$ describes the probability distribution of characteristic/property $A$ in a population $Ω$ .

Both interpretations are compatible with the set-theoretic model used throughout this document.

1.5 - Set Theory

Consider the sample space as the set of all outcomes in an experiment, and an event as a subset of the sample space.

As such, for any two events, we can take their union and intersection with the standard conventions of disjunction and conjunction, respectively.

We define the complement of an event $E$ as the set difference $E^{∁} = {Ω ∖ E}$ , such that $P (E^{∁}) = 1 - P (E)$ .

1.5.1 - Properties of Set Operations

involutivity: $(E^{∁})^{∁} = E$
commutativity: $E \cup F = F \cup E$ and $E \cap F = F \cap E$
associativity: $(E \cup F) \cup G = E \cup (F \cup G)$ and $(E \cap F) \cap G = E \cap (F \cap G)$
distributivity: $(E \cup F) \cap G = (E \cap G) \cup (F \cap G)$ and $(E \cap F) \cup G = (E \cup G) \cap (F \cup G)$

1.5.2 - Probability's Inclusion-Exclusion Principle

The inclusion-exclusion principle can be used to show that

P (A \cup B) = P (A) + P (B) - P (A \cap B)

which is equivalent to "removing the double-counted".

1.6 - Classifying Events

1.6.1 - Mutually Exclusive Events

Two events are mutually exclusive if they cannot occur simultaneously.

Formally, as sets of outcomes, mutually exclusive events are disjoint sets. This is equivalent to saying that events $E, F$ are mutually exclusive if $E \cap F = \emptyset$ .

1.6.2 - Independent Events

Let $E, F$ be events. We say they are independent events if and only if both

\begin{aligned} P (E ∣ F) & = P (E) \\ P (F ∣ E) & = P (F) \end{aligned}

which, by the definition of conditional probability, is equivalent to saying

P (E \cap F) = P (E) \cdot P (F)

when events $E, F$ are independent.

"Knowing that one event occurs gives no information to whether the other also occurs."

1.6.3 - Relationship between Mutual Exclusivity and Independence

Let $E, F$ be events.

Events $E, F$ are mutually exclusive events if and only if $(E \cap F) = \emptyset$ , i.e. $P (E \cap F) = 0$ .

Events $E, F$ are independent if and only if $P (E \cap F) = P (E) \cdot P (F)$ .

So mutually exclusive events $E, F$ are only independent if $P (E) = 0$ or $P (F) = 0$ .

Therefore, if events $E, F$ have nonzero probability and are mutually exclusive, then $E, F$ are necessarily dependent events.

1.7 - Conditional Probability

The conditional probability of event $A$ given it is known that conditioning event $B$ occurs is

P (A ∣ B) = \frac{P (A \cap B)}{P (B)}

1.7.1 - Law of Total Probability

Let ${A_{1}, A_{2}, \dots}$ be a set of events that are mutually exclusive and collectively exhaustive.
For any event $E$ ,

\begin{aligned} P (E) & = \sum_{i} P (E \cap A_{i}) \\ = \sum_{i} P (E ∣ A_{i}) \cdot P (A_{i}) \end{aligned}

This is known as the law of total probability.

1.7.2 - Bayes's Theorem

Given events $A, B$ with $P (B) \neq 0$ ,

P (A ∣ B) = \frac{P (B ∣ A) \cdot P (A)}{P (B)}

This is known as Bayes's theorem.

1.8 - Random Variables

A random variable is a function $X : Ω \to R$ which maps an outcome to a number, in a way sufficient for analysis. See meaningfulness

1.9 - Probability Distribution

A probability distribution maps the values or ranges of values of outcomes of a random variable to the respective probabilities of the events they represent. They are the probability mass function and probability density function for discrete and continuous random variables, respectively.