Conditional Probability

To motivate this note, I’ll pose the following problem:

Consider . What is ?

At first glance, the answer seems simple: it’s 1/2! Closer inspection reveals that the event is not measurable: the rationals have measure zero, hence from our high-school (perhaps even undergraduate) definition of conditional probability, the given probability should be undefined. However, the conditional probability does exist, and it equals 1/2.

Redefining Conditional Probability

To account for events of measure zero, we must first redefine . So far, we’ve defined this as , . We overcome the case where by defining conditional probability as a random variable that satisfies some properties. The intuition behind this is that if the conditional probability doesn’t exist by the previous definition, we can use expectation to say that it ‘might’ exist and be equal to some expected value. This also nicely sets us up for conditional expectation and variance, and the law of total expectation and law of total variance. Before formally defining conditional probability, Recall the following definitions from Random Variables:

Definition: A Sub- algebra of is a algebra such that

Definition: A Measurable Function between two measurable spaces is a function such that for every , . If , then is said to be -measurable

Definition: The -algebra generated by a function is the collection of all inverse images .

With these three definitions, we define the conditional probability function on a sub- algebra as follows:

Definition: The conditional probability is a -measurable random variable such that

In terms of integrals, we can rewrite this as

Defining the conditioning on a sub- algebra rather than an event or another random variable has the following benefits:

• If we want to condition on a random variable , then
• If we want to condition on multiple random variables , then where (can be extended to a countable sequence of random variables)

Therefore, this definition of conditional probability is the most general definition.

Existence and Uniqueness of Conditional Probability

From the integral definition, it may be obvious that conditional probability is unique up to a set of measure zero, that is, if , then can take any value. This leads to the problem of regularized conditional probability. Essentially, we define values for sets whose measure is zero such that the resulting conditional probability function is ‘nice’, ie measurable. This would need us to set up constructs such as a Polish Space and then use some functional analysis results, but we’ll not get into that in this article.

Existence can be proved in many ways: here, we use the Radon-Nikodym theorem. I won’t be proving the theorem, but I’ll quote it, along with some exposition, which should be sufficient.

Definition: A measure is said to be absolutely continuous with respect to another measure (both defined on the same -algebra) if for every measurable set , . We denote this as , and we say that dominates

Theorem (Radon-Nikodym): if and are defined on such that , then there exists a -measurable function such that for any ,

The function is called the radon-nikodym derivative of and is denoted as .

We now prove our claim

Claim: exists and is unique upto a set of probability 0

Proof: Define to simply be , where is simply restricted to , and

Note and by the radon-nikodym theorem, exists and is unique upto a set of probability 0. Note that defining like so also takes care of the definition, as

Conditional Expectation, and deriving Conditional Probability from Conditional Expectation

Conditional expectation is defined in much the same way: as a random variable satisfying the property . In a handwavy sense, conditional expectation is ‘the best approximation’ of the specified quantity, given the information we have from . There’s a very nice functional analysis proof of this which treats random variables as vectors in a functional space and then approximates them using expectation, but I’ll leave that out for now.

Most of the same properties as conditional probability apply to conditional expectation as well, just that in the proof of existence, since expectation can be negative and we need a positive value for it to be a measure, we define as and proceed similarly.

In several textbooks, conditional expectation is derived first, and conditional probability follows. This is because . Also, a neat proof involving functional analysis is used to prove the existence of conditional expectation: is the orthogonal projection of onto the closed subspace , and from results about hilbert space, we can prove this.

The Problem

Now that we have this definition, how do we apply it? Sadly, we can’t directly obtain the conditional probability from it (unless we evaluate the radon-nikodym derivative), and the best we can do is assume that the conditional probability is a reasonable random variable, and then verify that it satisfies the definition.

For the given problem, . This is because for to be -measurable, must have all the information we’re given, namely whether is rational or not, hence the specification of the sub- algebra as mentioned. You can also think of it from the best approximation perspective: we’re only given information about whether X is rational or not and about X’s magnitude, and both those quantities are expressed in this sub- algebra. Now, let . This is -measurable (the preimage of any borel set lies in ). Verifying the two integrals gives us

Since both these integrals are equal for all , we have .

References:

1. Rosenthal, Jeffrey S. A First Look at Rigorous Probability Theory. 2nd ed, World Scientific, 2006.
2. https://math.stackexchange.com/questions/2306986/uniqueness-of-conditional-expectation
3. https://stats.stackexchange.com/questions/395310/what-does-conditioning-on-a-random-variable-mean
4. https://www.stat.berkeley.edu/users/pitman/s205f02/lecture15.pdf (this contains the projection proof)
5. http://www.stat.cmu.edu/~arinaldo/Teaching/36752/S18/Notes/lec_notes_6.pdf (this too has the projection proof)
6. Billingsley, Patrick. Probability and Measure. 3rd ed, Wiley, 1995.