1 Model One

Consider $$\begin{array}{}\text{(1.1)}& y_t\stackrel{\mathrm{i}.\mathrm{n}.\mathrm{d}}{\sim }\mathcal{N}({\mu }_t,{\sigma }^2).\end{array}$$ Note that RHS depends on ${\displaystyle t}$ so ${\displaystyle y_t}$ 's distribution changes with ${\displaystyle t}$, so we can't say "iid".
Parameters here are ${\displaystyle {\mu }_1,\cdots ,{\mu }_n,{\sigma }^2}$.
Without regularization, MLE gets ${\displaystyle {\mu }_t=y_t,{\sigma }^2=0}$. So we define ${\displaystyle {\hat{\mu }}_t^{\mathrm{ridge}}(\lambda )}$ and ${\displaystyle {\hat{\mu }}_t^{\mathrm{lasso}}(\lambda )}$, which minimize $$\sum _{t=1}^n(y_t-{\mu }_t{)}^2+\lambda \sum _{t=2}^{n-1}(({\mu }_{t+1}-{\mu }_t)-({\mu }_t-{\mu }_{t-1}){)}^2$$ and $$\sum _{t=1}^n(y_t-{\mu }_t{)}^2+\lambda \sum _{t=2}^{n-1}|({\mu }_{t+1}-{\mu }_t)-({\mu }_t-{\mu }_{t-1})|.$$ We already know from here that $${\hat{\mu }}_t^{\mathrm{ridge}}(\lambda )=X{\hat{\beta }}^{\mathrm{ridge}}(\lambda ),{\hat{\mu }}_t^{\mathrm{lasso}}(\lambda )=X{\hat{\beta }}^{\mathrm{lasso}}(\lambda ),$$ where $$X=\left(\begin{array}{ccccc}1& 0& 0& \cdots & 0\\ 1& 1& 0& \cdots & 0\\ 1& 2& 1& \cdots & 0\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ 1& n-1& n-2& \cdots & 1\end{array}\right),\beta =\left(\begin{array}{c}{\beta }_0\\ {\beta }_1\\ ⋮\\ {\beta }_{n-1}\end{array}\right),$$ and ${\displaystyle {\hat{\beta }}^{\mathrm{ridge}}(\lambda ),{\hat{\beta }}^{\mathrm{lasso}}(\lambda )}$ respectively minimize $$||y-X\beta |{|}^2+\sum _{t=2}^{n-1}{\beta }_t^2,||y-X\beta |{|}^2+\sum _{t=2}^{n-1}|{\beta }_t|.$$

This model is an example of the "mean model" where focus is on ${\displaystyle {\mu }_t}$. The next two models will be examples of "variance models".

2 Model Two

Consider $$\begin{array}{}\text{(2.1)}& y_t\stackrel{\mathrm{i}.\mathrm{n}.\mathrm{d}}{\sim }\mathcal{N}(0,{\tau }_t^2).\end{array}$$ The likelihood is $$\begin{array}{}\text{(2.2)}& \prod _{t=1}^n\frac{1}{{\tau }_t}\exp (-\frac{y_t^2}{2{\tau }_t^2}).\end{array}$$ Then ${\displaystyle (y_1^2,\cdots ,y_n^2)}$ form the sufficient statistic. Under (2.1), $$y_t^2\stackrel{\mathrm{i}.\mathrm{n}.\mathrm{d}}{\sim }{\tau }_t^2{\chi }_1^2.$$ Note that density of ${\displaystyle {\tau }_t^2{\chi }_1^2}$ is proportional to $$\frac{1}{{\tau }_t^2}{\left(\frac{x}{{\tau }_t^2}\right)}^{-\frac{1}{2}}\exp (-\frac{x}{2{\tau }_t^2})=x^{-\frac{1}{2}}\frac{1}{{\tau }_t}\exp (-\frac{x}{2{\tau }_t^2}).$$ The likelihood of ${\displaystyle (y_1^2,\cdots ,y_n^2)}$ is thus $$\prod _{t=1}^n(y_t^2{)}^{-\frac{1}{2}}\frac{1}{{\tau }_t}\exp (-\frac{y_t^2}{2{\tau }_t^2}).$$ Dropping ${\displaystyle (y_t^2{)}^{-\frac{1}{2}}}$, we have (2.2).

The log-likelihood is $$\sum _{t=1}^n(-\log {\tau }_t-\frac{y_t^2}{2{\tau }_t^2})\propto \sum _{t=1}^n(\log {\tau }_t+\frac{y_t^2}{2{\tau }_t^2})=\sum _{t=1}^n({\alpha }_t+\frac{y_t^2}{2}{\mathrm{e}}^{-2{\alpha }_t}).$$ Here we use ${\displaystyle {\alpha }_t=\log {\tau }_t}$, because it removes the constraint ${\displaystyle {\tau }_t>0}$, and improves computational stability. A simple minimization without regularization gets ${\displaystyle {\alpha }_t=\log |y_t|\Rightarrow {\tau }_t^2=y_t^2}$. So we introduce regularization. Assume ${\displaystyle {\alpha }_t}$ is smooth: ${\displaystyle {\hat{\alpha }}_t^{\mathrm{ridge}}(\lambda ),{\hat{\alpha }}_t^{\mathrm{lasso}}(\lambda )}$, which minimizes $$\sum _{t=1}^n({\alpha }_t+\frac{y_t^2}{2}{\mathrm{e}}^{-2{\alpha }_t})+\lambda \sum _{t=2}^{n-1}(({\alpha }_{t+1}-{\alpha }_t)-({\alpha }_t-{\alpha }_{t-1}){)}^2$$ and $$\sum _{t=1}^n({\alpha }_t+\frac{y_t^2}{2}{\mathrm{e}}^{-2{\alpha }_t})+\lambda \sum _{t=2}^{n-1}|({\alpha }_{t+1}-{\alpha }_t)-({\alpha }_t-{\alpha }_{t-1})|.$$

3 Model Three

Apply model two with DFT. Recall that for ${\displaystyle y_0,\cdots ,y_{n-1}}$, its DFT is ${\displaystyle b_0,\cdots ,b_{n-1}}$, where $$b_j=\sum _{t=0}^{n-1}{y}_t\exp (-\frac{2\pi ijt}{n}).$$ Assume ${\displaystyle n}$ is odd, ${\displaystyle m=\frac{n-1}{2}}$. So we obtain model three from model two for the DFT terms ${\displaystyle b_1,\cdots ,b_m}$, and assume $$\mathrm{Re}(b_j),\mathrm{Im}(b_j)\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }\mathcal{N}(0,{\gamma }_j^2),j=1,\cdots ,m.$$ Also assume ${\displaystyle b_j}$ are independent across ${\displaystyle j}$. The unknown parameters here are ${\displaystyle {\gamma }_1,\cdots ,{\gamma }_m}$. ${\displaystyle {\gamma }_j}$ represents the strength of sinusoids at frequency ${\displaystyle j/n}$. The likelihood is $$\begin{array}{rl}& \prod _{j=1}^m\frac{1}{{\gamma }_j}\exp (-\frac{(\mathrm{Re}(b_j){)}^2}{2{\gamma }_j^2})\frac{1}{{\gamma }_j}\exp (-\frac{(\mathrm{Im}(b_j){)}^2}{2{\gamma }_j^2})\\ =& \prod _{j=1}^m\frac{1}{{\gamma }_j^2}\exp (-\frac{(\mathrm{Re}(b_j){)}^2+(\mathrm{Im}(b_j){)}^2}{2{\gamma }_j^2})=\prod _{j=1}^m\frac{1}{{\gamma }_j^2}\exp (-\frac{|b_j{|}^2}{2{\gamma }_j^2}).\end{array}$$
Recall: periodogram ${\displaystyle I(j/n)}$ is defined as $$I(j/n)=\frac{|b_j{|}^2}{n}.$$ We can therefore rewrite the likelihood: $$\prod _{j=1}^m\frac{1}{{\gamma }_j^2}\exp (-\frac{nI(j/n)}{2{\gamma }_j^2}).$$ So the periodogram forms the sufficient statistic in this model. Further $$I(j/n)=\frac{1}{n}|b_j{|}^2=\frac{1}{n}((\mathrm{Re}(b_j){)}^2+(\mathrm{Im}(b_j){)}^2)\sim \frac{{\gamma }_j^2}{n}{\chi }_2^2.$$ So we rewrite the model as $$I(j/n)\stackrel{\mathrm{i}.\mathrm{n}.\mathrm{d}}{\sim }\frac{{\gamma }_j^2}{n}{\chi }_2^2,j=1,\cdots ,m.$$ The negative log-likelihood is $$\sum _{j=1}^m(2\log {\gamma }_j+\frac{nI(j/n)}{2{\gamma }_j^2})=\sum _{j=1}^m(2{\alpha }_j+\frac{nI(j/n)}{2}{\mathrm{e}}^{-2{\alpha }_j}),$$ where ${\displaystyle {\alpha }_j=\log {\gamma }_j}$. Directly minimize it, we have $${\alpha }_j=\log \sqrt{\frac{nI(j/n)}{2}},{\gamma }_j^2={\mathrm{e}}^{2{\alpha }_j}=\frac{nI(j/n)}{2}.$$ Next regularize it: ${\displaystyle {\hat{\alpha }}_t^{\mathrm{ridge}}(\lambda )}$, ${\displaystyle {\hat{\alpha }}_t^{\mathrm{lasso}}(\lambda )}$: $$\sum _{j=1}^m(2{\alpha }_j+\frac{nI(j/n)}{2}{\mathrm{e}}^{-2{\alpha }_j})+\lambda \sum _{j=2}^{m-1}(({\alpha }_{j+1}-{\alpha }_j)-({\alpha }_j-{\alpha }_{j-1}){)}^2,$$ and $$\sum _{j=1}^m(2{\alpha }_j+\frac{nI(j/n)}{2}{\mathrm{e}}^{-2{\alpha }_j})+\lambda \sum _{j=2}^{m-1}|({\alpha }_{j+1}-{\alpha }_j)-({\alpha }_j-{\alpha }_{j-1})|.$$