8 MGF, Convolution

#MGF #Convolution #GeometricDistribution #ExponentialDistribution #Moments #BernoulliProcess #GammaDistribution #LaplaceDistribution #Convergence #NormalDistribution #PoissonDistribution #BinomialDistribution #NegativeBinomialDistribution

1 MGF

MGF

The moment generating function (MGF) of a RV $X$ is a function $M_{X} : R \to [0, + \infty)$ given by $M_{X} (t) = E [e^{t X}], t \in R .$

If $X_{1}, \dots, X_{n}$ are independent, $S_{n} = X_{1} + \dots + X_{n}$ , then $M_{S_{n}} (t) = \prod_{k = 1}^{n} M_{X_{k}} (t) .$

We can show quickly that $M_{α X + β} (t) = E [e^{(α X + β) t}] = e^{β t} E [e^{α X t}] = e^{β t} M_{X} (α t) .$

MGF is very important in calculating moments in that

Theorem

If $M_{X} (t) < \infty, \forall t \in (- ε, ε)$ , then

${\frac{d^{k}}{d t^{k}} M_{X} (t) |}_{t = 0} = E [X^{k}]$ .
For $t$ within the radius of convergence, $M_{X} (t) = \sum_{k = 0}^{\infty} E [X^{k}] \frac{t^{k}}{k!}$ .

Theorem (Uniqueness)

Suppose $X, Y$ are two RVs with well defined MGFs. If $M_{X} (t) = M_{Y} (t), \forall t \in (- ε, ε)$ , then $X \overset{d}{=} Y$ .

Recall discussion on convergence in distribution.

Theorem (Convergence in Distribution/Continuity Theorem)

Suppose $X_{1}, X_{2}, \dots$ is a sequence of RVs with MGF $M_{X_{n}} (t)$ well defined for $t \in (- ε, ε)$ . If $M_{X_{n}} (t) \to M (t), \forall t \in (- ε, ε)$ , then $M (t) = M_{X} (t)$ , where $M_{X}$ is the MGF of a RV $X$ s.t. $X_{n} \overset{d}{\to} X$ .

Application 1:

\frac{1}{n} Geometric (\frac{λ}{n}) \overset{d}{\to} Exp (λ)

Let $G \sim Geometric (p)$ . By definiton of Geometric distribution, $P (G = k) = (1 - p)^{k - 1} p, k \in N^{*}$ . And $E [G] = \frac{1}{p}$ , so $\begin{aligned} M_{G} (t) & = E [e^{t G}] = \sum_{k = 1}^{\infty} e^{t k} (1 - p)^{k - 1} p \\ = p e^{t} \sum_{k = 1}^{\infty} [e^{t} (1 - p)]^{k - 1} = \frac{p e^{t}}{1 - (1 - p) e^{t}} . \end{aligned}$
Let $G_{n} \sim Geometric (\frac{λ}{n})$ , where $λ > 0$ . Let $X_{n} = \frac{G_{n}}{n}$ , then $E [X_{n}] = \frac{1}{λ}$ . So $M_{X_{n}} (t) = E [e^{t X_{n}}] = \frac{\frac{λ}{n} e^{\frac{t}{n}}}{1 - (1 - \frac{λ}{n}) e^{\frac{t}{n}}} \to \frac{λ}{λ - t} .$
Let $X \sim Exp (λ)$ , $λ > 0$ . Then $X$ has p.d.f $f_{X} (x) = {\begin{aligned} λ e^{- λ x}, x \geq 0, \\ 0, x < 0. \end{aligned}$ And $E [X] = \frac{1}{λ}, M_{X} (t) = E [e^{t X}] = \int_{0}^{\infty} e^{t X} λ e^{- λ x} d x = \frac{λ}{λ - t}$ .
So $X_{n} \overset{d}{\to} X$ .

Recall Bernoulli process: $P (success) = p, P (failure) = 1 - p$ . $T_{r}$ is total number of trials until the $r$ th success. $F_{r}$ is total number of failures until the $r$ th success. $F_{r} + r = T_{r}$ . We showed that $F_{r} \sim NegativeBinomial (r, p)$ . And $T_{r} = W_{1} + \dots + W_{r}$ , where $W_{1}, \dots, W_{r} \overset{i . i . d}{\sim} Geometric (p)$ . So by application 1, $M_{T_{r}} (t) = {[\frac{p e^{t}}{1 - (1 - p) e^{t}}]}^{r} .$
So $M_{F_{r}} (t) = E [e^{t F_{r}}] = e^{- t r} E [e^{t T_{r}}] = {[\frac{p}{1 - (1 - p) e^{t}}]}^{r} .$
Let $X_{n} = \frac{F_{r, n}}{n}$ , where $F_{r, n} \sim NegativeBinomial (r, \frac{λ}{n})$ , then $M_{X_{n}} (t) = E [e^{t \frac{F_{r, n}}{n}}] = [\frac{\frac{λ}{n}}{1 - (1 - \frac{λ}{n}) e^{\frac{t}{n}}}] \to {(\frac{λ}{λ - t})}^{r} .$
Now let $X = Y_{1} + \dots + Y_{r}$ , where $Y_{1}, \dots, Y_{r} \overset{i . i . d}{\sim} Exp (λ)$ . Then $X \sim Gamma (r, λ)$ , with p.d.f $f_{X} (x) = \frac{λ^{r}}{Γ (r)} x^{r - 1} e^{- λ x},$ and $M_{X} (t) = \prod_{i = 1}^{n} M_{Y_{i}} (t) = {(\frac{λ}{λ - t})}^{r} .$
So $X_{n} \overset{d}{\to} X$ .

Application 3:

N (μ_{1}, σ_{1}^{2}) + N (μ_{2}, σ_{2}^{2}) \overset{d}{\to} N (μ_{1} + μ_{2}, σ_{1}^{2} + σ_{2}^{2})

We have shown in this proof that the MGF of $G \sim N (μ, σ^{2})$ is given by $M_{G} (t) = E [e^{t X}] = e^{μ t + \frac{σ^{2} t^{2}}{2}} .$
So if $X_{1} \sim N (μ_{1}, σ_{1}^{2}), X_{2} \sim N (μ_{2}, σ_{2}^{2})$ with $X_{1} ⊥ ⊥ X_{2}$ , then $M_{X_{1} + X_{2}} (t) = M_{X_{1}} (t) M_{X_{2}} (t) = e^{(μ_{1} + μ_{2}) t + (σ_{1}^{2} + σ_{2}^{2}) \frac{t^{2}}{2}},$ this is also MGF of $N (μ_{1} + μ_{2}, σ_{1}^{2} + σ_{2}^{2})$ . By uniqueness of MGF, $X_{1} + X_{2} \sim N (μ_{1} + μ_{2}, σ_{1}^{2} + σ_{2}^{2})$ .

Application 4:

Binomial (n, \frac{λ}{n}) \overset{d}{\to} Poisson (λ)

Let $Y_{n} \sim Binomial (n, p_{n})$ , where $p_{n} = \frac{λ}{n}$ . Then $Y_{n} = I_{1} + \dots + I_{n}$ , where $I_{1}, \dots, I_{n} \overset{i . i . d}{\sim} Bernoulli (p_{n})$ . So $M_{Y_{n}} (t) = [p_{n} e^{t} + (1 - p_{n})]^{n} .$ By a result from mathematical analysis: $lim_{n \to \infty} a_{n} = a \Rightarrow lim_{n \to \infty} {(1 + \frac{a_{n}}{n})}^{n} = e^{a},$ we have $lim_{n \to \infty} M_{Y_{n}} (t) = {[1 + \frac{λ}{n} (e^{t} - 1)]}^{n} \to e^{λ (e^{t} - 1)} .$
Now for $Y \sim Poisson (λ), λ > 0$ , $M_{Y} (t) = E [e^{t Y}] = \sum_{k = 0}^{\infty} e^{t k} \frac{λ^{k} e^{- λ}}{k!} = e^{- λ} \sum_{k = 0}^{\infty} \frac{1}{k!} (λ e^{t})^{k} = e^{- λ} e^{λ e^{t}} .$
So $Y_{n} \overset{d}{\to} Y$ .

2 Convolution

Convolution can be applied to analyze sum of random variables.
Let $X, Y$ be discrete RVs on the same probability space. We want to find $P (X + Y = c)$ . Then $\begin{aligned} P (X + Y = c) & = \sum_{(a, b) : a + b = c} P (X = a, Y = b) \\ = \sum_{a} P (X = a, Y = c - a) . \end{aligned}$
We assume $X ⊥ ⊥ Y$ , then $P (X + Y = c) = \sum_{a} P (X = a) P (Y = c - a) .$
This is called convolution.

Example

$X_{i} \sim Geometric (p_{i}), i = 1, 2$ , $X_{1} ⊥ ⊥ X_{2}$ . For $c = 2, 3, \dots$ , $\begin{aligned} P (X_{1} + X_{2} = c) & = \sum_{a = 1}^{c - 1} P (X_{1} = a) P (X_{2} = c - a) \\ = \sum_{a = 1}^{c - 1} (1 - p_{1})^{a - 1} p_{1} (1 - p_{2})^{c - a - 1} p_{2} \\ = (\frac{p_{2}}{p_{2} - p_{1}}) p_{1} (1 - p_{1})^{c - 1} + (\frac{p_{1}}{p_{1} - p_{2}}) p_{2} (1 - p_{2})^{c - 1} . \end{aligned}$

For continuous RVs, similarly we have $f_{X + Y} (z) = \int_{- \infty}^{+ \infty} f_{X} (x) f_{Y} (z - x) d x .$

Example

For $i = 1, 2$ , $X_{i} \sim Exp (λ_{i}), λ_{i} > 0, λ_{1} \neq λ_{2}$ . $X_{1} ⊥ ⊥ X_{2}$ . For $z > 0$ , $\begin{aligned} f_{X_{1} + X_{2}} (z) & = \int_{- \infty}^{+ \infty} f_{X_{1}} (x) f_{X_{2}} (z - x) d x \\ = \int_{0}^{z} f_{X_{1}} (x) f_{X_{2}} (z - x) d x \\ = \int_{0}^{z} λ_{1} e^{- λ_{1} x} λ_{2} e^{- λ_{2} (z - x)} d x \\ = λ_{1} λ_{2} e^{- λ_{2} z} {\frac{e^{- x (λ_{1} - λ_{2})}}{- (λ_{1} - λ_{2})} |}_{0}^{z} \\ = (\frac{λ_{2}}{λ_{2} - λ_{1}}) λ_{1} e^{- λ_{1} z} + (\frac{λ_{1}}{λ_{1} - λ_{2}}) λ_{2} e^{- λ_{2} z} . \end{aligned}$

We use another example to show the application of both MGF and convolution.

Example: Difference of Exponential is Laplace

Let $X, Y \overset{i . i . d}{\sim} Exp (λ)$ . Prove that $Z = X - Y$ follows a Laplace distribution.

Use MGF.
Calculate the MGF of $Z$ : $\begin{aligned} M_{Z} (t) & = \iint_{R^{2}} e^{t (x - y)} f_{X, Y} (x, y) d x d y \\ = \int_{- \infty}^{+ \infty} e^{t X} f_{X} (x) d x \int_{- \infty}^{+ \infty} e^{- t y} f_{Y} (y) d y \\ = λ \int_{0}^{+ \infty} e^{(t - λ) x} d x \cdot λ \int_{0}^{+ \infty} e^{- (t + λ) y} d y = \frac{λ^{2}}{λ^{2} - t^{2}}, | t | < λ . \end{aligned}$
Then for $- λ < t < λ$ , MGF is identical to that of Laplace distribution.
Use convolution.
$f_{X - Y} (z) = \int_{- \infty}^{+ \infty} f_{X} (x) f_{Y} (x - z) d x$ .
For $z > 0$ , $\begin{array}{r} f_{X - Y} (z) = \int_{z}^{+ \infty} λ^{2} e^{- λ x} e^{- λ (x - z)} d x = \frac{λ}{2} e^{- λ z} . \end{array}$
For $z < 0$ , $f_{X - Y} (z) = \int_{0}^{+ \infty} λ^{2} e^{- λ x} e^{- λ (x - z)} d x = \frac{λ}{2} e^{λ z} .$
In conclusion, $f_{X - Y} (z) = \frac{λ}{2} e^{- λ | z |}$ , so $Z$ follows a Laplace distribution.

Note that the concrete expression is $NegativeBinomial (r, \frac{λ}{n}) / n$ . ↩︎