1 Distribution Derived from 0-1 Sequence

Consider a sequence of ${\displaystyle 0}$ s and ${\displaystyle 1}$ s. (call this Bernoulli Process) Denote each index a RV ${\displaystyle X_i,i=1,\cdots ,n}$.

1.1 Bernoulli Distribution

If ${\displaystyle X\sim \mathrm{Bernoulli}(p)}$, then ${\displaystyle \mathbb{P}(X=1)=p,\mathbb{P}(X=0)=1-p}$. ${\displaystyle (0<p<1)}$
${\displaystyle \mathbb{E}[X]=p,\mathrm{Var}(X)=\mathbb{E}[X^2]-\mathbb{E}[X{]}^2=p(1-p)}$.

If ${\displaystyle X_1,\cdots ,X_n\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }\mathrm{Bernoulli}(p)}$, then call this a Bernoulli process. Let ${\displaystyle S_n=X_1+\cdots +X_n}$. This leads to:

1.2 Binomial Distribution

${\displaystyle S_n\sim \mathrm{Binomial}(n,p)}$, then ${\displaystyle \mathbb{P}(S_n=k)=(\genfrac{}{}{0ex}{}{n}{k})p^k(1-p{)}^{n-k}}$.
By linearity of expectation, $$\mathbb{E}[S_n]=\mathbb{E}\left[\sum _{i=1}^n{X}_i\right]=\sum _{i=1}^n\mathbb{E}[X_i]=np,$$
By independence of ${\displaystyle X_1,\cdots ,X_n}$ and (3.1), $$\mathrm{Var}(S_n)=\sum _{i=1}^n\mathrm{Var}(X_i)=np(1-p).$$

1.3 Geometric Distribution

Denote ${\displaystyle W_i}$ as the waiting time between the ${\displaystyle (i-1)}$ th and ${\displaystyle i}$ th successes. Then ${\displaystyle W_i\sim \mathrm{Geometric}(p)}$. ${\displaystyle \mathbb{P}(W=k)=(1-p{)}^{k-1}p,k\in {\mathbb{N}}^{\ast }}$.
Then $$\begin{array}{rl}\mathbb{E}[W]& =\sum _{k=1}^{\mathrm{\infty }}k(1-p{)}^{k-1}p=\frac{1}{p},\\ \mathrm{Var}(W)& =\mathbb{E}[W^2]-\mathbb{E}[W{]}^2=\sum _{k=1}^{\mathrm{\infty }}{k}^2(1-p{)}^{k-1}p-{\left(\frac{1}{p}\right)}^2=\frac{1-p}{p^2}.\end{array}$$

These moments can be computed more easily using MGF.
By here we know the MGF for ${\displaystyle W\sim \mathrm{Geometric}(p)}$ is $$M_W(t)=\frac{p{\mathrm{e}}^t}{1-(1-p){\mathrm{e}}^t},$$ so $$\begin{array}{rl}\mathbb{E}[W]& ={\frac{\mathrm{d}}{\mathrm{d}t}{M}_W(t)|}_{t=0}={\frac{p{\mathrm{e}}^t}{(1-(1-p){\mathrm{e}}^t{)}^2}|}_{t=0}=\frac{1}{p},\\ \mathbb{E}[W^2]& ={\frac{{\mathrm{d}}^2}{\mathrm{d}{t}^2}{M}_W(t)|}_{t=0}={\frac{p{\mathrm{e}}^t}{(1-(1-p){\mathrm{e}}^t{)}^3}|}_{t=0}=\frac{1}{p^2}.\end{array}$$

1.4 Negative Binomial Distribution

${\displaystyle r\in {\mathbb{N}}^{\ast }}$. Use ${\displaystyle T_r}$ to denote the total waiting time to the ${\displaystyle r}$ th success. Then ${\displaystyle T_r=W_1+\cdots +W_r}$, ${\displaystyle W_1,\cdots ,W_r\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }\mathrm{Geometric}(p)}$. Then $$\mathbb{P}(T_r=n)=(\genfrac{}{}{0ex}{}{n-1}{r-1})p^{r-1}(1-p{)}^{n-r}\cdot p.$$
Use ${\displaystyle F_r}$ to denote number of failures before ${\displaystyle r}$ th success, then ${\displaystyle F_r+r=T_r}$. Denote ${\displaystyle F_r\sim \mathrm{NegativeBinomial}(r,p)}$. $$\mathbb{P}(F_r=k)=(\genfrac{}{}{0ex}{}{r+k-1}{r-1})p^r(1-p{)}^k=(\genfrac{}{}{0ex}{}{r+k-1}{k})p^r(1-p{)}^k.$$
Then $$\begin{array}{rl}\mathbb{E}[F_r]& =\mathbb{E}[T_r]-r=\frac{r}{p}-r=\frac{r(1-p)}{p},\\ \mathrm{Var}(F_r)& =\mathrm{Var}(T_r-r)=\mathrm{Var}(T_r)=r\mathrm{Var}(W_1)=\frac{r(1-p)}{p^2}.\end{array}$$

2 Hypergeometric Distribution

There are ${\displaystyle B}$ blue balls and ${\displaystyle R}$ red balls. ${\displaystyle N=B+R}$. Sample size ${\displaystyle n<N}$.

If sample ${\displaystyle n}$ balls u.a.r with replacement, let ${\displaystyle X(\omega )}$ be the number of blue balls in ${\displaystyle \omega \in \mathrm{\Omega }}$ (sample space). Then ${\displaystyle X\sim \mathrm{Binomial}(n,p)}$, ${\displaystyle p=\frac{B}{N}}$.

If sample ${\displaystyle n}$ balls u.a.r without replacement, let ${\displaystyle X(\omega )}$ be number of blue balls. Obviously, when ${\displaystyle k>B}$ or ${\displaystyle n-k>R}$, $$\mathbb{P}(X=k)=0.$$
For ${\displaystyle max\{0,n-R\}\le k\le B}$, consider ${\displaystyle \omega }$ as an event that ${\displaystyle X=k}$. Denote $$q=\mathbb{P}(\{\omega \})=\frac{\frac{B!}{(B-k)!}\frac{R!}{(R-(n-k))!}}{\frac{N!}{(N-n)!}}.$$

• @ Key observation: for another event ${\displaystyle {\omega }^{\prime }}$ with the same condition, the probability is the same.

So there are ${\displaystyle (\genfrac{}{}{0ex}{}{n}{k})}$ distinct sequences with ${\displaystyle k}$ blue balls and ${\displaystyle n-k}$ red balls. So $$\mathbb{P}(X=k)=(\genfrac{}{}{0ex}{}{n}{k})q=\frac{(\genfrac{}{}{0ex}{}{B}{k})(\genfrac{}{}{0ex}{}{N-B}{n-k})}{(\genfrac{}{}{0ex}{}{N}{n})}.$$
This distribution is called Hypergeometric Distribution, ${\displaystyle X\sim \mathrm{Hypergeometric}(N,B,n)}$.

${\displaystyle \mathbb{P}(X=k)\to (\genfrac{}{}{0ex}{}{n}{k})p^k(1-p{)}^{n-k}}$ as ${\displaystyle B\to \mathrm{\infty },N\to \mathrm{\infty },\frac{B}{N}\to p}$.

2.1 Expectation

Define indicator RV ${\displaystyle I_i(\omega )=\{\begin{array}{rl}& 1,i\text{-th draw is blue for }\omega \in \mathrm{\Omega },\\ & 0,\text{otherwise}.\end{array}}$
Then ${\displaystyle X=I_1+\cdots +I_n}$, ${\displaystyle (I_1,\cdots ,I_n)}$ is exchangeable. So ${\displaystyle \mathbb{P}(I_i=1)=\mathbb{P}(I_1=1)=\frac{B}{N}=\mathbb{E}[I_1]}$, so $$\mathbb{E}[X]=\sum _{i=1}^n\mathbb{E}[I_i]=n\mathbb{E}[I_1]=\frac{nB}{N}.$$

This is the same as expectation of ${\displaystyle \mathrm{Binomial}(n,\frac{B}{N})}$.

2.2 Variance

By exchangeability,

Since $$\begin{array}{rl}\mathrm{Var}(I_1)& =\frac{B}{N}(1-\frac{B}{N}),\\ \mathrm{Cov}(I_1,I_2)& =\mathbb{E}[I_1{I}_2]-\mathbb{E}[I_1]\mathbb{E}[I_2],\\ \mathbb{E}[I_1{I}_2]& =\mathbb{P}(I_1=I_2=1)=\frac{B}{N}\left(\frac{B-1}{N-1}\right),\end{array}$$
then $$\mathrm{Cov}(I_1,I_2)=-\frac{B(N-B)}{N^2(N-1)}<0,$$ and $$\mathrm{Var}(X)=n\frac{B}{N}\left(\frac{N-B}{N}\right)\left(\frac{N-n}{N-1}\right).$$

Yellow part is the variance of ${\displaystyle \mathrm{Binomial}(n,\frac{B}{N})}$; green part ${\displaystyle \le 1}$.

3 Multinomial Distribution

Consider repeated trials. Each with ${\displaystyle m}$ types of outcome. (For Bernoulli trials, ${\displaystyle m=2}$; for rolling a dice, ${\displaystyle m=6}$)
Denote ${\displaystyle N}$ as number of trials, ${\displaystyle X_i}$ as number of times type ${\displaystyle i}$ is observed.
Denote ${\displaystyle (X_1,\cdots ,X_m)=\overrightarrow{X}}$ (${\displaystyle X_1,\cdots ,X_m}$ are not independent given ${\displaystyle N=n}$), then ${\displaystyle (X_1,\cdots ,X_m)|N=n\sim \mathrm{Multinomial}(n,p_1,\cdots ,p_m)}$, with ${\displaystyle p_j\in [0,1],\sum _{j=1}^m{p}_j=1}$.
A possible outcome ${\displaystyle \overrightarrow{a}=(a_1,\cdots ,a_m)}$ should satisfy ${\displaystyle \sum _{j=1}^m{a}_j=n,a_j\in \{0,\cdots ,n\}}$. So $$\mathbb{P}(\overrightarrow{X}=\overrightarrow{a}|N=n)=(\genfrac{}{}{0ex}{}{n}{a_1,\cdots ,a_m})p_1^{a_1}\cdots p_m^{a_m},$$ where ${\displaystyle (\genfrac{}{}{0ex}{}{n}{a_1,\cdots ,a_m})=\frac{n!}{a_1!\cdots a_m!}}$.

Poissonization of the Multinomial
Now suppose number of trials is a RV ${\displaystyle N\sim \mathrm{Poisson}(\lambda )}$. We want to determine distribution of ${\displaystyle (X_1,\cdots ,X_m)}$. So $$\begin{array}{rl}\mathbb{P}(\overrightarrow{X}=\overrightarrow{a})& =\sum _{n=0}^{\mathrm{\infty }}\mathbb{P}(\overrightarrow{X}=\overrightarrow{a}|N=n)\mathbb{P}(N=n)\\ & =\sum _{n=0}^{\mathrm{\infty }}\frac{n!}{a_1!\cdots a_m!}{p}_1^{a_1}\cdots p_m^{a_m}\mathbf{1}\{\sum _{j=1}^m{a}_j=n\}\cdot \frac{{\lambda }^n}{n!}{\mathrm{e}}^{-\lambda }\\ & =\frac{p_1^{a_1}\cdots p_m^{a_m}}{a_1!\cdots a_m!}{\lambda }^{a_1+\cdots +a_m}{\mathrm{e}}^{-\lambda (p_1+\cdots +p_m)}\\ & =\left[\frac{(p_1\lambda {)}^{a_1}}{a_1!}{\mathrm{e}}^{-p_1\lambda }\right]\cdots \left[\frac{(p_m\lambda {)}^{a_m}}{a_m!}{\mathrm{e}}^{-p_m\lambda }\right].\end{array}$$
So if ${\displaystyle N\sim \mathrm{Poisson}(\lambda )}$,

1. ${\displaystyle X_1,\cdots ,X_m}$ are independent.
2. ${\displaystyle X_j\sim \mathrm{Poisson}(p_j\lambda )}$,

Then $$X\sim \mathrm{Multinomial}(N,p_1,\cdots ,p_m).$$