HW4

Due Friday 09/15/2017 11:55:00 PM

Please try to complete before class on Thursday in case there are questions or clarifications needed (or post on Piazza). Use LaTeX or write by hand (must be legible) and scan to submit via Sakai.

Consider the linear model $Y = X\beta + \epsilon$ with $E[\epsilon] = 0_n$ and Cov$(\epsilon) = \sigma^2 I_n$ and with $X$ of full column rank $p$. Answer the following questions. Hint: rewrite each as a quadratic form

1. Consider estimation of $\beta$ using quadratic loss $(\beta- a)^T(\beta- a)$ for some estimator $a$. Find the expected quadratic loss if we use the MLE $\hat{\beta}$ for $a$ conditional on $X$. Simplify the expression as a function of the eigenvalues of $X^TX$. What happens as the smallest eigenvalue of $X^TX$ goes to $0$?

2. Consider estimation of $\mu$ at the observed data points $X$. Find the expected quadratic loss
   $E[(\mu - X\hat{\beta})^T(\mu - X\hat{\beta})]$ conditional on $X$. What happens as the smallest eigenvalue of $X^TX$ goes to 0?

3. Consider predicting a new $Y_{*}$ at the observed data points $X$ where $Y_{*}$ is independent of $Y$. Find the expected quadratic loss for $E[(Y_{*} - X\hat{\beta})^T(Y_{*} - X\hat{\beta})]$. What happens as the smallest eigenvalue of $X^TX$ goes to 0?

4. Consider predicting $Y_{*}$’s at new points $X_{*}$ with $E[X_{*}^TX_{*}] = I_p$. Find the expected quadratic loss $E[(Y_{*} - X_{*}\hat{\beta})^T(Y_{*} - X_{*}\hat{\beta})]$ conditional on $X$ and $X_{*}$ and then unconditional on $X_{*}$ (but still conditional on $X$. What happens as the smallest eigenvalue of $X^TX$ goes to 0? (If $E[X_{*}^TX_{*}] = \Sigma > 0$ does that change the result)

5. Comment on the difference in estimation and prediction at observed data versus new data as $X$ becomes non-full rank. Which is the most stable? Which is the least?

Review Chapter 2 and Appendix C in Plane Answers to Complex Questions