1 Change of Slope Model

For ${\displaystyle t\le c}$, the slope of the regression line is ${\displaystyle {\beta }_1}$, while for ${\displaystyle t>c}$, is ${\displaystyle {\beta }_1+{\beta }_2}$. The parameters here are ${\displaystyle c,{\beta }_0,{\beta }_1,{\beta }_2,\sigma }$.

If ${\displaystyle c}$ were known, then (1.1) is a linear: $$y=X_c\beta +c,$$ here ${\displaystyle X_c=\left(\begin{array}{ccc}1& 1& \mathrm{ReLU}(1-c)\\ ⋮& ⋮& ⋮\\ 1& n& \mathrm{ReLU}(1-c)\end{array}\right)}$.

2 Parameter Estimation

2.1 MLE

We also use MLE: $$\mathrm{RSS}(c)=\underset{{\beta }_0,{\beta }_1,{\beta }_2}{min}\sum _{t=1}^n(y_t-{\beta }_0-{\beta }_{1}t-{\beta }_2\mathrm{ReLU}(t-c){)}^2.$$ After numerically solving this, the model becomes linear as mentioned just now, so $$\hat{\beta }=({\hat{\beta }}_0,{\hat{\beta }}_1,{\hat{\beta }}_2{)}^{\mathrm{T}}=(X_{\hat{c}}^{\mathrm{T}}{X}_{\hat{c}}{)}^{-1}{X}_{\hat{c}}^{\mathrm{T}}y.$$ Then $$\hat{\sigma }=\sqrt{\frac{\mathrm{RSS}(\hat{c})}{n-3}}.$$

2.2 Uncertainty Quantification

Now switch to Bayes. Suppose prior: $${\beta }_0,{\beta }_1,{\beta }_2,\log \sigma \stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }\mathrm{Unif}(-C,C)$$ for a large ${\displaystyle C}$. For ${\displaystyle c}$, it is also uniform and the range is ${\displaystyle 1,\cdots ,n}$. However, ${\displaystyle c}$ can't be ${\displaystyle 1}$ or ${\displaystyle n}$: when ${\displaystyle c=1}$, $${\beta }_0+{\beta }_{1}t+{\beta }_2\mathrm{ReLU}(t-1)={\beta }_0+{\beta }_{1}t+{\beta }_2(t-1)$$ is linear. Same for ${\displaystyle c=n}$. Therefore $$c\sim \mathrm{Unif}\{2,\cdots ,n-1\}.$$
Then $$\pi (c|\mathrm{data})\propto {\left(\frac{1}{\mathrm{RSS}(c)}\right)}^{\frac{n-3}{2}}|X_c^{\mathrm{T}}{X}_c{|}^{-\frac{1}{2}}\mathbf{1}\{c=2,\cdots ,n-1\}.$$ Given ${\displaystyle c}$, the model is linear. By Homework 1 Problem 4, $$\frac{\mathrm{RSS}(c)}{{\sigma }^2}|\mathrm{data},c\sim {\chi }_{n-3}^2.$$ Finally, $$\beta |\mathrm{data},c,\sigma \sim {\mathcal{N}}_3({\hat{\beta }}_c,{\sigma }^2(X_c^{\mathrm{T}}{X}_c{)}^{-1}),{\hat{\beta }}_c=(X_c^{\mathrm{T}}{X}_c{)}^{-1}{X}_c^{\mathrm{T}}y.$$

2.3 Posterior Sampling for Uncertainty Quantification

Drawing posterior samples from unknown parameters is a useful way to visualize the uncertainty.

If we want to obtain posterior samples for ${\displaystyle y_{t^{\ast }}}$ for future ${\displaystyle t^{\ast }}$, then based on the algorithm, in step 2, add 2.5: generate ${\displaystyle y_{t^{\ast }}^{(j)}\sim \mathcal{N}({\beta }_0^{(j)}+{\beta }_1^{(j)}{t}^{\ast }+{\beta }_2^{(j)}\mathrm{ReLU}(t^{\ast }-c^{(j)},({\sigma }^{(j)}{)}^2)}$.

3 More Change of Slope

If we want to introduce one more break point: $$y_t={\beta }_0+{\beta }_{1}t+{\beta }_2\mathrm{ReLU}(t-c_1)+{\beta }_3\mathrm{ReLU}(t-c_2)+{\epsilon }_t,$$ with ${\displaystyle {\epsilon }_t\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }\mathcal{N}(0,{\sigma }^2)}$, we can also write it as $$y=X_c\beta +\epsilon ,X_c=\left(\begin{array}{cccc}1& 1& \mathrm{ReLU}(1-c_1)& \mathrm{ReLU}(1-c_2)\\ ⋮& ⋮& ⋮& ⋮\\ 1& n& \mathrm{ReLU}(n-c_1)& \mathrm{ReLU}(n-c_2)\end{array}\right).$$ Then we can also have posterior $$\pi (c|\mathrm{data})\propto {\left(\frac{1}{\mathrm{RSS}(c)}\right)}^{\frac{n-4}{2}}|X_c^{\mathrm{T}}{X}_c{|}^{-\frac{1}{2}}.$$ For more break points, we can consider $$y_t={\beta }_0+{\beta }_{1}t+\sum _{j=1}^k{\beta }_{j+1}\mathrm{ReLU}(t-c_j)+{\epsilon }_t.$$