1. Let \(P\) be an orthogonal projection matrix (symmetric and idempotent). Show the following:

    1. \(I - P\) is idempotent;
    2. \(P a\) is orthogonal to \(( I - P) b\) for any vectors \(a\) and \(b\);
    3. \(y = P y + ( I - P )y\);
    4. \(|| y||^2 = || P y||^2 + ||( I- P) y||^2\).

  2. Let \(E \in \mathbb R^{n\times p}\) and let \(S=E^\top E\).

    1. Show that all eigenvalues of \(S\) are non-negative.
    2. If \(v\) is an eigenvector of \(S\) with \(Sv=0\), what is the relationship between the rows of \(E\) and \(v\)?
    3. Show that if \(S v_1 = \lambda_1 v_1\) and \(S v_2 = \lambda_2 v_2\) with \(\lambda_1\neq \lambda_2\) then \(v_1^\top v_2 = 0\).
    4. If Suppose the eigenvalues of \(S\) are distinct, and let \(V \in \mathbb R^{p\times p}\) have columns equal to the unit eigenvectors of \(S\). Show that \(V^\top V = V V^\top = I_p\).

  3. Suppose the best one-dimensional affine approximation to the rows of \(Y\in \mathbb R^{n\times p}\) is \(\hat Y = 1 \bar y^\top + f_1 v_1^\top\), where \(v_1^\top v_1 = 1\). For \(i=1,\ldots,n\), let \(x_i = a + c B (y_i-\bar y)\), where \(y_i\) is the \(i\)th row of \(Y\), \(a\in \mathbb R^p\), \(B\) is an orthogonal matrix (\(B^\top B=I_p)\) and \(c\) is a scalar. Let \(\hat X = 1 m^\top + g w^\top\) be the best one-dimensional affine approximation to \(X\).

    1. Express \(m\), \(g\) and \(w\) in terms of \(a\), \(B\), \(c\), \(\bar y\), \(f_1\) and \(v_1\).
    2. Explain conditions on \(a\), \(B\) and \(c\) for which \(m=\bar y\).
    3. Explain conditions on \(a\), \(B\) and \(c\) for which \(w=v_1\).

  4. Let \(Y\) be the data matrix on Swiss head measurements discussed in class.

    1. Compute the first two eigevectors \(v_1\) and \(v_2\) of the sample covariance matrix of \(Y\), and interpret the coefficients.
    2. Let \(X\) be the \(200\times 6\) matrix with row \(i\) equal to \(x_i = y_i/(1_p^\top y_i)\), so the sum of each row of \(X\) is 1. Compute the first eigevector of the sample covariance matrix of \(X\), and compare to \(v_1\) and \(v_2\) from part a. Explain what you notice.