HW 8 Solutions

STA 211 Spring 2023 (Jiang)

Exercise 1

Consider a simple linear regression model given by \(Y_i = X_i\beta + \epsilon_i\) for \(i = 1, \cdots, n\), and \(\epsilon_i\) are i.i.d. and distributed \(N(0, \sigma^2)\) where \(\sigma^2\) is known (note that there is no intercept in this model). Find the MLE of \(\beta\). How does it compare to the OLS estimator?

Note that each \(Y_i\) is i.i.d. distributed with \(N(X_i\beta, \sigma^2)\).

\[\begin{align*} \mathcal{L} (\beta) &= \prod_{i = 1}^n f_Y(y_i)\\ &= \prod_{i = 1}^n \frac{i = 1}{n} \frac{1}{\sqrt{2\pi\sigma^2}}\exp\left(-\frac{(Y_i - X_i\beta)}{2\sigma^2} \right)\\ &= (2\pi\sigma^2)^{-n/2}\exp\left(-\frac{\sum_{i = 1}^n(Y_i - X_i\beta)^2}{2\sigma^2}\right)\\ \log\mathcal{L} (\beta) &= -\frac{n}{2}\log(2\pi) - \frac{n}{2}\log(\sigma^2) - \frac{1}{2\sigma^2}\sum_{i = 1}^n (Y_i - X_i\beta)^2\\ \frac{\partial}{\partial \beta}\log\mathcal{L} (\beta) &= \frac{\partial}{\partial \beta}\left( - \frac{1}{2\sigma^2}\sum_{i = 1}^n (Y_i - X_i\beta)^2 \right)\\ &= \frac{\partial}{\partial \beta}\left( - \frac{1}{2\sigma^2}\sum_{i = 1}^n Y_i^2 - 2X_i\beta Y_i + (X_i\beta)^2 \right)\\ &= \frac{1}{\sigma^2}\sum_{i = 1}^n X_iY_i - \frac{1}{\sigma^2}\sum_{i = 1}^n X_i^2\beta \end{align*}\]

Setting equal to 0 and solving for a candidate MLE:

\[\begin{align*} \sum_{i = 1}^n X_i^2\beta &= \sum_{i = 1}^n X_iY_i\\ \widehat{\beta} &= \frac{\sum_{i = 1}^n X_iY_i}{\sum_{i = 1}^n X_i^2} \end{align*}\]

Verifying that it is indeed the MLE:

\[\begin{align*} \frac{\partial}{\partial \beta}\left(\frac{1}{\sigma^2}\sum_{i = 1}^n X_iY_i - \frac{1}{\sigma^2}\sum_{i = 1}^n X_i^2\beta\right) &= -\frac{1}{\sigma^2}\sum_{i = 1}^n X_i^2\\ &< 0. \end{align*}\]

For the ordinary least squares estimator, note the following for a vector \(X\) (these are all scalar quantities):

\[\begin{align*} X^TX &= \sum_{i = 1}^n X_i^2\\ X^TY &= \sum_{i = 1}^n X_iY_i \end{align*}\]

Thus

\[\begin{align*} (X^TX)^{-1}X^TY &= \frac{\sum_{i = 1}^n X_iY_i}{\sum_{i = 1}^n X_i^2}\\ &= \widehat{\beta} \end{align*}\]

Exercise 2

Suppose \(Y_1, \cdots, Y_n\) are an i.i.d. sample from some distribution where \(f_Y(y) = \theta y^{-\theta + 1}\) for \(y > 1\) and \(\theta > 0\). What is the MLE for \(\theta\)?

Note that each \(Y_i\) is i.i.d. distributed with \(f_Y(y) = \theta y^{-\theta + 1}\) (for appropriate values of \(y\) and \(\theta\)).

\[\begin{align*} \mathcal{L}(\theta) &= \prod_{i = 1}^n f_Y(y_i)\\ &= \prod_{i = 1}^n \theta y_i^{-\theta + 1}\\ &= n\theta \prod_{i = 1}^n y_i^{-\theta + 1}\\ \log \mathcal{L}(\theta) &= n\log(\theta) + (1 - \theta) \sum_{i = 1}^n \log(y_i)\\ \frac{\partial}{\partial \theta}\log\mathcal{L} (\theta) &= \frac{n}{\theta} - \sum_{i = 1}^n \log (y_i) \end{align*}\]

Setting equal to 0 and solving for a candidate MLE:

\[\begin{align*} \frac{n}{\theta} &= \sum_{i = 1}^n \log (y_i)\\ \widehat{\theta} &= \frac{n}{\sum_{i = 1}^n \log (y_i)} \end{align*}\]

Verifying that it is indeed the MLE:

\[\begin{align*} \frac{\partial}{\partial \theta}\left( \frac{n}{\theta} - \sum_{i = 1}^n \log (y_i)\right) &= -n\theta^{-2}\\ &< 0. \end{align*}\]

Exercise 3

Suppose \(Y_1, \cdots, Y_n\) are an i.i.d. sample from some distribution where \(f_Y(y) = \theta(1 - \theta)^{y - 1}\) where \(y\) takes on positive integer values and \(0 < \theta < 1\). What is the MLE for \(\theta^{-1}\)?

Note that each \(Y_i\) is i.i.d. distributed with \(f_Y(y) = \theta(1 - \theta)^{y - 1}\) (for appropriate values of \(y\) and \(\theta\)).

\[\begin{align*} \mathcal{L}(\theta) &= \prod_{i = 1}^n f_Y(y_i)\\ &= \prod_{i = 1}^n \theta(1 - \theta)^{y_i - 1}\\ &= \theta^n (1 - \theta)^{\sum_{i = 1}^n y_i - n}\\ \log \mathcal{L}(\theta) &= n\log (\theta) + \left(\sum_{i = 1}^n y_i - n\right)\log(1 - \theta)\\ \frac{\partial}{\partial \theta}\log \mathcal{L}(\theta) &= \frac{n}{\theta} - \frac{\sum_{i = 1}^n y_i - n}{1 - \theta} \end{align*}\]

Setting equal to 0 and solving for a candidate MLE for \(\theta\):

\[\begin{align*} \frac{n}{\theta} &= \frac{\sum_{i = 1}^n y_i - n}{1 - \theta}\\ \widehat{\theta} &= \frac{n}{\sum_{i = 1}^n y_i} \end{align*}\]

Verifying that it is indeed the MLE for \(\theta\):

\[\begin{align*} \frac{\partial}{\partial \theta}\left( \frac{n}{\theta} - \frac{\sum_{i = 1}^n y_i - n}{1 - \theta}\right) &= -n\theta^{-2} - \frac{\sum_{i = 1}^n y_i - n}{(1 - \theta)^2}\\ &< 0. \end{align*}\]

Thus the MLE for \(\theta^{-1}\) is \[\frac{1}{n}\sum_{i = 1}^n y_i.\]