1 - What is Probability?

1.1 - Experiments

An experiment is a procedure, subject to uncertainty, which necessarily produces an outcome.

A collection of outcomes is called an event.
The collection of all outcomes is called the sample space.
The collection of all events to be considered is the event space.

An event is said to occur if it contains the observed outcome.
Probability concerns events and how likely they are to occur.

1.2 - Probability Axioms

Probability is axiomatized by the probability axioms:

1. the probability of an event is a non-negative real number
2. the probability of the entire sample space is 1
3. any mutually exclusive events have probability which is additive

1.3 - Probability as a Measure

A model is a mathematical tool which systematizes a set of axioms to something which can be manipulated. See probability space for the rigorous model in the most technical language.

1.3.1 - Definitions

Here, we define

• the sample space $\mathrm{\Omega }$ is the set of all outcomes,
• an event $E\subseteq \mathrm{\Omega }$ is a subset of outcomes,
• the set of all events $\mathcal{F}=\{E:E\subseteq \mathrm{\Omega }\}$ is the event space
  Then the probability measure $P:\mathcal{F}\to [0,1]$ is a measure (a type of function) which takes an event $E\in \mathcal{F}$ and assigns it a real number $P(E)\in [0,1]$.

1.3.2 - Interpreting the Axioms

Now, the probability axioms can be rewritten

1. $P(E)\in [0,1]$ for all events $E$
2. $P(\mathrm{\Omega })=1$
3. $P(\bigcup E_i)=\sum P(E_i)$ for mutually exclusive events $E_i$

1.3.3 - Corollaries to the Axioms

1. $P(\varnothing )=0$
2. $E\subseteq F⟹P(E)\le P(F)$

1.4 - Competing Interpretations

Plainly: For each event $A$, there is some associated probability $P(A)$.

Under the frequentist interpretation of probability, $P(A)$ is the frequency with which $A$ would occur under a theoretically infinite number of repetitions of the experiment (i.e. the law of large numbers)

Under the statistical interpretation of probability, $P(A)$ describes the probability distribution of characteristic/property $A$ in a population $\mathrm{\Omega }$.

Both interpretations are compatible with the set-theoretic model used throughout this document.

1.5 - Set Theory

Consider the sample space as the set of all outcomes in an experiment, and an event as a subset of the sample space.

As such, for any two events, we can take their union and intersection with the standard conventions of disjunction and conjunction, respectively:

• complement $A^{\complement }$ is all outcomes not in $A$
• union $A\cup B$ is all outcomes in either $A$ or $B$
• intersection $A\cap B$ is all outcomes in both $A$ and $B$

Explicitly, the complement of an event $E$ is the set difference $E^{\complement }=\{\mathrm{\Omega }\setminus E\}$, such that $P(E^{\complement })=1-P(E)$.

1.5.1 - Properties of Set Operations

• involutivity: $(E^{\complement }{)}^{\complement }=E$
• commutativity: $E\cup F=F\cup E$ and $E\cap F=F\cap E$
• associativity: $(E\cup F)\cup G=E\cup (F\cup G)$ and $(E\cap F)\cap G=E\cap (F\cap G)$
• distributivity: $(E\cup F)\cap G=(E\cap G)\cup (F\cap G)$ and $(E\cap F)\cup G=(E\cup G)\cap (F\cup G)$

1.5.2 - Probability's Inclusion-Exclusion Principle

The inclusion-exclusion principle can be used to show that

which is equivalent to "removing the double-counted".

1.6 - Classifying Events

1.6.1 - Mutually Exclusive Events

Two events are mutually exclusive if they cannot occur simultaneously.

Formally, as sets of outcomes, mutually exclusive events are disjoint sets. This is equivalent to saying that events $E,F$ are mutually exclusive if $E\cap F=\varnothing$.

1.6.2 - Independent Events

Let $E,F$ be events. We say they are independent events if and only if both

which, by the definition of conditional probability, is equivalent to saying

when events $E,F$ are independent. We may write $E\perp F$.

"Knowing that one event occurs gives no information to whether the other also occurs."

"The occurrence of either event does not affect the occurrence of the other."

1.6.3 - Relationship between Mutual Exclusivity and Independence

Let $E,F$ be events.

Events $E,F$ are mutually exclusive events if and only if $(E\cap F)=\varnothing$, i.e. $P(E\cap F)=0$.

Events $E,F$ are independent if and only if $P(E\cap F)=P(E)\cdot P(F)$.

So mutually exclusive events $E,F$ are only independent if $P(E)=0$ or $P(F)=0$.

Therefore, if events $E,F$ have nonzero probability and are mutually exclusive, then $E,F$ are necessarily dependent events.

1.6.4 - Collectively Exhaustive Events

A collection of events $\{A_1,\dots ,A_n\}$ is said to be collectively exhaustive if $A_1\cup \cdots \cup A_n=\mathrm{\Omega }$, i.e. the events together cover the sample space.

1.7 - Conditional Probability

The conditional probability of event $A$ given that conditioning event $B$ occurs is expressed by

1.7.1 - Law of Total Probability

Let $\{A_1,A_2,\dots \}$ be a set of events that are mutually exclusive and collectively exhaustive.
For any event $E$,

This is known as the law of total probability.

1.7.2 - Multiplication Rule

This is known as the multiplication rule.

1.7.3 - Reducing the Sample Space

If $E\subseteq F$, then $E\cap F=E$, so $P(E\mid F)=\frac{P(E)}{P(F)}$.
Since $P(F)\in [0,1]$, then $P(E\mid F)\ge P(E)$.
This is called "reducing the sample space"

1.7.4 - Bayes's Theorem

Given events $A,B$ with $P(B)\ne 0$,

This is known as Bayes's theorem.

1.8 - Random Variables

A random variable is a function $X:\mathrm{\Omega }\to \mathbb{R}$ which maps each outcome to a quantity, in a way sufficient for analysis. See meaningfulness

The support of a real-valued function is the subset of the domain which is not mapped to zero.

A discrete random variable has a support which is finite or countably infinite.
A continuous random variable has a range which is an interval of real numbers.

1.9 - Probability Distribution

A probability distribution maps the values or ranges of values of outcomes of a random variable to the respective probabilities of the events they represent.

1.9.1 - Probability Mass Function

A probability mass function is a probability distribution which maps every value of a discrete random variable to its associated probability.

Formally,
a discrete random variable $X:\mathrm{\Omega }\to \mathbb{R}$ has a probability mass function $p:\mathbb{R}\to [0,1]$ such that

i.e. $p(x)$ is the probability that $X$ takes a value of exactly $x$.

1.9.2 - Probability Density Function

A probability density function is a probability distribution which maps every interval of values of a continuous random variable to its associated probability.

Formally,
a continuous random variable $X:\mathrm{\Omega }\to \mathbb{R}$ has probability density function $f(x)$ such that for any two $a\le b$ in $\mathbb{R}$,

where $f(x)$ is non-negative and ${\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}f(x)dx=1$.

1.9.3 - Cumulative Distribution Function

A cumulative distribution function is a probability distribution which returns the probability that the random variable will take a value of at most the argument of the function.

Formally,
a random variable $X$ has a cumulative distribution function $F:\mathbb{R}\to [0,1]$ such that

i.e. $F(x)$ is the probability that $X$ will take a value of at most $x$.

For a discrete random variable with probability mass function $p(x)$, the cumulative distribution function is given by

For a continuous random variable with probability density function $f(x)$, the cumulative distribution function is given by

1.10 - Measures

1.10.1 - Expected Value

The expected value of a random variable is the average value it will take when weighted by the probability of each outcome, aka the mean. It is a measure of center.

Formally,
the expected value of a discrete random variable $X$ with probability mass function $p(x)$ is given by

and the expected value of a continuous random variable $X$ with probability density function $f(x)$ is given by

1.10.2 - Variance

The variance of a random variable is a measure of dispersion from the mean.

Formally,
the variance of a random variable $X$ is given by

where $E$ is the expected value and $\mu$ is the mean $\mu =E[X]$.

This can be simplified to

1.10.3 - Standard Deviation

The standard deviation of a random variable is a measure of dispersion from the mean. It is the square root of the variance, and has the same units as its data.

Formally,
the standard deviation $\sigma$ of a random variable $X$ is given by

where $\mathrm{Var}(X)=E[(X-\mu {)}^2]$ is the variance, and $\mu =E[X]$ is the mean.