Statistics & Gene Expression Data Analysis Note 6: SVD in Multiple Regression

♠ Multiple Linear Regression Model

• Recall the model in matrix form:

y = X′β + ϵ

• Predictor variables in p × n matrix X = [x_{1}, x_{2}, ⋯, x_{n}] (columns are samples, rows are predictor variables)

♠ SVD of X

• SVD is X = BF or X = ADF

where F = [f_{1}, f_{2}, …, f_{n}] is n × n matrix of factors (columns represent samples, and rows represent factor variables)

♠ SVD Regression

• Combine the SVD with the regression model to get

y = F′θ + ϵ

with

• θ = B′β or θ = DA′β

• Multiple regression on the factor variables themselves as predictors

• n predictor variables, not p

• Regression parameter vector θ to estimate

• Dimension reduction of inference/estimation problem when p > n, as is the case in gene expression analyses

♠ Bayesian Analysis and Stochastic Regularisation

• If p > n we end up with n parameters to be estimated with n observations

• Least squares and other standard methods inapplicable: exact fit to observed data, no predictive value (“over-fitting”)

• Generally, remove some factors that do not vary or contribute much to the SVD (small values of the singular values in the D matrix)

• More useful and formal solutions lie in Bayesian analysis that involves “stochastic regularisation” of the

estimation problem – estimate θ with some partial constraints on values imposed probabilistically

(Insert two semesters of statistics in here please!).

• Typically, reduce to a smaller number of factors and then apply Bayesian analysis to the rest

• Corresponding estimation of β via β = AD^{−1}θ

♠ Software, Computation and Summary

• Point estimate analysis: iterative computation of estimates of θ that are Bayesian posterior modes (EM algorithms, MAP estimation)

• Full Bayesian analysis using stochastic simulation methods (Markov chain Monte Carlo simulation, Gibbs sampling): see discussion in the binary regression context

Mike West  June 2, 2004  Page