1. More linear algebra:
    1. Let \(P\) be a possibly oblique projection of \(\mathbb R^n\) onto \(C(X)\). Find a LUE \(\check{\beta}\) of \(\beta\) such that \(X\check{\beta} = P y\) for each \(y \in \mathbb R^n\), and show that it is unique.
    2. Suppose \(E[y]=X\beta\) and \(V[y] = \Sigma = X \Psi X^\top + \Omega\). Show that the GLS estimator \(\hat\beta_\Sigma\) is equal to the LUE given by \(\hat\beta_\Omega = (X^\top\Omega^{-1} X)^{-1} X^\top \Omega^{-1} y\).
  2. Weights: Consider the simple straight-line through the origin regression model, \(y_i = \beta x_i + \epsilon_i\) where \(\beta\) and \(x_i\) are scalars.
    1. Show that \(\hat \beta_w = \sum w_i y_i x_i/ \sum w_i x_i^2\) is an unbiased estimator of \(\beta\) as long as \(\sum w_i x_i^2 \neq 0\).
    2. Now suppose the \(\epsilon_i\)’s are uncorrelated but \(V[\epsilon_i] = \sigma^2 \times v_i\). Compute the variance of \(\hat\beta_w\) , and using calculus or some other method, find the values of \(w_1,\ldots, w_n\) that minimize this variance.
  3. Model misspecification: Suppose \(E[y] = \mu\) and \(V[y] = \sigma^2 I\). However, suppose we assume \(E[y]=X\beta\) and obtain the corresponding OLS estimate \(\hat \beta\) and unbiased estimate \(s^2\) of \(\sigma^2\).
    1. Compute \(E[\hat\beta]\) and \(E[X \hat\beta]\) in terms of \(\mu\), \(X\) and the projection matrix \(P\) onto \(C(X)\). Describe the possible bias of \(X\hat\beta\) as an estimator of \(\mu\). What is \(\hat\beta\) measuring?
    2. Compute \(E[ \hat\epsilon ]\) and \(E[s^2]\) and describe its possible bias as an estimator of \(\sigma^2\). What is \(s^2\) measuring?
    3. Now suppose that \(\mu = W\alpha+ X\beta\) for some matrix \(W\in \mathbb R^{n\times q}\), \(\alpha\in \mathbb R^q\) and \(\beta\in \mathbb R^p\), but we are unaware of \(W\) and just fit a model assuming \(\mu=X\beta\). Repeat items (a) and (b) above in this case, and identify conditions under which \(\hat\beta\) is unbiased for \(\beta\), and conditions under which \(s^2\) is unbiased for \(\sigma^2\). Can both \(\hat\beta\) and \(s^2\) be unbiased?
  4. Simulation study: Let \(X\) be a \(5\times 2\) matrix with columns \((1,1,1,1,1)\) and \((1,2,3,4,5)\). For each \(r\in \{ 1,2,4\}\) do the following:
    1. Row-bind \(r-1\) copies of \(X\) to itself to create the \(n\times 2\) matrix \(X_r\) with \(n=5\times r\), and compute the \(n\times 1\) vector \(\mu\) where \(\mu=X_r\beta\), with \(\beta=(1/10,1/5)\).
    2. Simulate 1000 \(\hat\beta\) and \(s^2\) values by repeating the following 1000 times: Simulate \(y_i \sim Poisson(\mu_i)\) independently for \(i=1,\ldots,n\) and compute the OLS regression estimate \(\hat\beta\) the residual variance \(s^2\).
    3. Describe the distribution of \(\hat\beta\) and \(s^2\) graphically, using histograms, scatterplots or other figures. Comment on the effect of \(r\) on these distributions.
    4. Consider an approximate 95% confidence interval for \(\beta_2\) (the “slope”) of the form \(\hat\beta_2 \pm 1.96 s c\), where \(c\) is an appropriate value obtained from the normal approximation to the distribution of \(\hat\beta\). Explain in writing how you obtain \(c\), and compute this interval for each simulated dataset. Report the fraction of datasets for which the interval covers the true value of \(1/5\).
  5. GLS and feasible GLS: Suppose \(y_1,\ldots, y_n\) are independent exponential random variables with \(E[y_i] = 1+3\times x_i\). Using a numerical study, compare the MSE and bias of the OLS, GLS and FGLS estimators of \(\beta=(1,3)\) as follows: Let \(X\) be a \(10\times 2\) matrix with columns \((1,\ldots,1)\) and \((1,\ldots,10)\). For each \(r\in \{ 1,2,4\}\) do the following:
    1. Row-bind \(r-1\) copies of \(X\) to itself to create the \(n\times 2\) matrix \(X_r\) with \(n=10\times r\), and compute the \(n\times 1\) vector \(\mu\) where \(\mu=X_r\beta\), with \(\beta=(1,3)\).
    2. Write out the formula for the variance-covariance matrix for the OLS and GLS estimators in the model \(E[y]=\mu\) with each \(y_i\) being exponential, compute them numerically, and compare them.
    3. Simulate 1000 datasets from the model, and compute the OLS, GLS, and FGLS estimators for each.
    4. Compute the empirical approximation to the matrix \(E[ (\hat\beta_{FGLS} - \beta)(\hat\beta_{FGLS} - \beta)^\top]\) and compare it to the variance-covariance matrices of the OLS and GLS estimators. Does FGLS work in this case? Is the FGLS estimator biased?
    5. Make a scatterplot of the simulated estimators, and describe how their concentrations around the true \(\beta\) differ.