Probability Theory

Probability Theory#

Probability theory underpins every foundational machine learning development we will be treating in this course. We start by defining two simple and intuitive rules from which basically every other concept is derived from.

Rules of probability#

Consider we are picking at random a very large number $N$ of balls from a bag, from which two variables $X$ and $Y$ with possible values $x_{i}$ and $y_{j}$ can be observed (for instance the color and material of each ball).

The grid below shows all possible “bins” our draws can fall into, with five possible values for $X$ and four possible values for $Y$ :

../../../../../_images/probgrid.svg — Fig. 1 Two random variables, with possible values on a grid and some sampled observations (gray)#

You can each gray circle above to be one of the $N$ times we sample $X$ and $Y$ . From this figure, we can define some very important concepts:

Joint probability

$p (X = x_{i}, Y = y_{j}) = \frac{n_{i j}}{N}$

The probability of observing $x_{i}$ and $y_{j}$ together.

Marginal probability

$p (X = x_{i}) = \sum_{j = 1}^{4} p (X = x_{i}, Y = y_{j}) = \frac{c_{i}}{N}$ The probability of observing $x_{i}$ regardless of $Y$ .

$p (Y = y_{j}) = \sum_{i = 1}^{5} p (X = x_{i}, Y = y_{j}) = \frac{r_{j}}{N}$ The probability of observing $y_{j}$ regardless of $X$ .

Conditional probability

$p (Y = y_{j} | X = x_{i}) = \frac{n_{i j}}{c_{i}}$ The probability of observing $y_{j}$ given that $X = x_{i}$ .

$p (X = x_{i} | Y = y_{j}) = \frac{n_{i j}}{r_{j}}$ The probability of observing $x_{i}$ given that $Y = y_{j}$ .

Which can all be observed from the shaded regions in the figure above.

From these three observations, we can get to the two crucial rules of probability, namely the sum and the product rules:

Sum rule

$p (X) = \sum_{Y} p (X, Y)$

From the joint, we can marginalize variables by summing over all other variables.

Product rule

$p (X, Y) = p (Y | X) p (X)$

$p (X, Y) = p (X | Y) p (Y)$

We can recover the joint probability by combining a marginal and a conditional.

Probability densities#

We now move to expressing probabilities over continuous variables. In that case, it does not make sense anymore to have a probability associated with any given value of the variable, as there are infinitely many possibilities.

Instead, we can only compute the probability that a value falls somewhere within a range. To do this, we introduce a probability density $p (x)$ and express the probability that $x$ falls within a given interval $[a, b]$ as:

(1)#

p (x \in [a, b]) = \int_{a}^{b} p (x) d x

Furthermore, probability densities are always subject to the two constraints:

(2)#

p (x) \geq 0

(3)#

\int_{- \infty}^{\infty} p (x) d x = 1

which makes intuitive sense: probabilities cannot be negative, and once every possibility is accounted for they should sum up to 1.

The same rules of probability are also valid for probability densities, that is:

Rules of probability for densities

$p (x) = \int p (x, y) d y$ Sum rule

$p (x, y) = p (y | x) p (x)$ Product rule

Expectations, Monte Carlo approximation#

Consider a function $f (x)$ applied to a random variable with probability density $p (x)$ . Since the value of $x$ is uncertain, it stands to reason that the value of $f$ should also vary. The average value of $f$ is defined as its expectation:

Expectation of a function

$E [f] = \int f (x) p (x) d x$

Sometimes we do not have an exact expression for $p (x)$ but we can somehow get samples from it, think for instance of performing a series of experimental measurements. In this case we can approximate the expectation of $f$ in a Monte Carlo way:

Monte Carlo approximation

$E [f] \approx \frac{1}{N} \sum_{i}^{N} f (x_{i})$

Variances and covariances#

Again for a function $f (x)$ of a random variable $x$ , it is also useful to compute how spread out the response is around the expected value. This is done by computing its variance:

Variance of a function

$var [f] = E [{(f (x) - E [f (x)])}^{2}]$

Through a similar argument we can compute the covariance between two random variables $x$ and $y$ :

Covariance between two variables

$cov [x, y] = E [(x - E [x]) (y - E [y])]$

which expresses how strong the correlation between $x$ and $y$ is.

If the covariance between $x$ and $y$ is high, we can feel confident in observing only one of them and inferring the other since we can reasonably expect them to change together.