High-dimensional Regression with Sinusoids

Traditional sinusoidal models cannot capture important characteristics of the sunspots dataset. We can fix this by including sinusoid terms for all possible ${\displaystyle f}$. We stick to Fourier frequencies. Say ${\displaystyle n}$ is odd, then consider ${\displaystyle \frac{1}{n},\frac{2}{n},\cdots ,\frac{n-1}{2n}}$. Denote $$\begin{array}{}\text{(1.1)}& y_t={\beta }_0+\sum _{j=1}^m({\beta }_{1j}\cos \left(2\pi \frac{j}{n}t\right)+{\beta }_{2j}\sin \left(2\pi \frac{j}{n}t\right))+{\epsilon }_t,\end{array}$$ where ${\displaystyle {\epsilon }_t\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }\mathcal{N}(0,{\sigma }^2)}$. We can write this model in matrix form as ${\displaystyle y=X\beta +\epsilon }$, where $$X=\left(\begin{array}{cccccc}1& \cos (2\pi (1/n)1)& \sin (2\pi (1/n)1)& \cdots & \cos (2\pi (m/n)1)& \sin (2\pi (m/n)1)\\ ⋮& ⋮& ⋮& \ddots & ⋮& ⋮& \\ 1& \cos (2\pi (1/n)n)& \sin (2\pi (1/n)n)& \cdots & \cos (2\pi (m/n)n)& \sin (2\pi (m/n)n)\end{array}\right),$$ where ${\displaystyle m=\frac{n-1}{2}}$, and ${\displaystyle \beta =({\beta }_0,{\beta }_{11},{\beta }_{21},\cdots ,{\beta }_{1m},{\beta }_{2m}{)}^{\mathrm{T}}\in {\mathbb{R}}^{n\times 1}}$.

Without regularization, if we only want to minimize ${\displaystyle ||y-X\beta |{|}^2}$, we can get a perfect fit $$\begin{array}{rl}{\hat{\beta }}_0& =\bar{y},\\ {\hat{\beta }}_{1k}& =\frac{2}{n}\sum _{t=1}^n{y}_t\cos \left(2\pi \frac{k}{n}t\right),\\ {\hat{\beta }}_{2k}& =\frac{2}{n}\sum _{t=1}^n{y}_t\sin \left(2\pi \frac{k}{n}t\right).\end{array}$$
To prevent overfitting, we can use ridge regularization: $$\sum _{t=1}^n{(y_t-{\beta }_0-\sum _{j=1}^m({\beta }_{1j}\cos \left(2\pi \frac{j}{n}t\right)+{\beta }_{2j}\sin \left(2\pi \frac{j}{n}t\right)))}^2+\lambda \sum _{j=1}^m({\beta }_{1j}^2+{\beta }_{2j}^2).$$
From discuss here, we can also understand it as $$\begin{array}{}\text{(1.2)}& {\beta }_{11},{\beta }_{21},\cdots ,{\beta }_{1m},{\beta }_{2m}\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }\mathcal{N}(0,{\tau }^2).\end{array}$$

The Spectrum Model

The spectrum model is based on (1.1) and (1.2), and has two changes:

• ${\displaystyle {\epsilon }_t}$ is removed;
• Not assuming identically distributed.

Therefore we have $$\begin{array}{}\text{(2.1)}& y_t={\beta }_0+\sum _{j=1}^m({\beta }_{1j}\cos \left(2\pi \frac{j}{n}t\right)+{\beta }_{2j}\sin \left(2\pi \frac{j}{n}t\right)),\end{array}$$ with $${\beta }_{11},{\beta }_{21}\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }\mathcal{N}(0,{\tau }_1^2),\cdots ,{\beta }_{1m},{\beta }_{2m}\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }\mathcal{N}(0,{\tau }_m^2).$$
The parameter ${\displaystyle ({\tau }_1^2,\cdots ,{\tau }_m^2)}$ is known as the spectrum of the model. Then $$\begin{array}{rl}\mathrm{Var}(y_t)& =\sum _{j=1}^m\mathrm{Var}({\beta }_{1j}\cos \left(2\pi \frac{j}{n}t\right)+{\beta }_{2j}\sin \left(2\pi \frac{j}{n}t\right))\\ & =\sum _{j=1}^m[\mathrm{Var}({\beta }_{1j}){\cos }^2\left(2\pi \frac{j}{n}t\right)+\mathrm{Var}({\beta }_{2j}){\sin }^2\left(2\pi \frac{j}{n}t\right)]\\ & =\sum _{j=1}^m{\tau }_j^2.\end{array}$$
(${\displaystyle {\tau }_j^2}$ represents the contribution of frequency ${\displaystyle j/n}$ to the overall ${\displaystyle \mathrm{Var}(y_t)}$).

For ${\displaystyle {\beta }_0}$, average both sides of (2.1), we have ${\displaystyle {\beta }_0=\bar{y}}$. I.e. $$y_t-\bar{y}=\sum _{j=1}^m({\beta }_{1j}\cos \left(2\pi \frac{j}{n}t\right)+{\beta }_{2j}\sin \left(2\pi \frac{j}{n}t\right)),{\beta }_{1j},{\beta }_{2j}\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }\mathcal{N}(0,{\tau }_j^2).$$

So data generated from this model look differently depending on ${\displaystyle {\tau }_1^2,\cdots ,{\tau }_m^2}$.