Statistics & Gene Expression Data Analysis Note 3: Multiple Regression

♠ Multiple linear regression models

• Extend straight line regression model to use more than one predictor gene (dependent variable) response gene y (e.g., ER) (independent variable, explanatory variable) predictor genes x_{1}, x_{2}, ..., x_{p}

• Measurement error model: repeat values i = 1, …, n, - independent expression levels on n tumors

y_{i} = α + ∑_{r=1}^{p} β_{r}x_{r,i} + ϵ_{i}

x_{r,i} is expression of gene r on array i

• Model “explains” variability in response y “due to” p genes

• Non-causal, purely empirical

• Predictive validity: fit model and test in new cases

• Interpretation: β_{r} measures change in expected response with a unit change in predictor x_{r}

• Value and interpretation of β_{r} depends, sometimes critically, on which other genes/predictors are in specified model

• Analysis and inference:

- Estimate parameters (α, β_{1}, ..., β_{p}, σ^{2})

- Predict new (“future”) responses ...

♠ Notation: Matrices and vectors

• Intercept term α = β_{0} = β_{0}x_{0} with x_{0} = 1 (“dummy” gene with constant expression)

• Revise earlier notation for x_{i}, X to include dummy/intercept

• (p + 1) × 1 column vector

x_{i} = ( 1 ; x_{1,i} ; x_{2,i} ; ⋮ ; x_{p,i} )

• Regression parameter vector

β = ( β_{0} ; β_{1} ; ⋮ ; β_{p} )

• Model is y_{i} = x′_{i}β + ϵ_{i}

• Expression data in (p + 1) × n matrix X = [x_{1}, x_{2}, ⋯, x_{n}]

• Response variable and errors in n × 1 vectors

y = ( y_{1} ; y_{2} ; ⋮ ; y_{n} ) and ϵ = ( ϵ_{1} ; ϵ_{2} ; ⋮ ; ϵ_{n} )

• Model in matrix form:

y = X′β + ϵ

♠ Least squares fitting

• For any chosen β,

Q(β) = ∑_{i=1}^{n} ϵ_{i}^{2} = ∑_{i=1}^{n}(y_{i} − x′_{i}β)^{2}

measures “fit” of chosen line x_{i}β to response data

• Choose β̂ to minimise Q(β)

• Least squares estimates (LSE)

• Fitted least squares line: ŷ = X′β̂

• Residuals: e = y − ŷ with elements e_{i} = y_{i} − ŷ_{i} – what is left ‘unexplained’ in response data

– estimates of ϵ_{i}

♠ LSE formulæ:

•

β̂ = VXy and V = (XX′)^{−1}

(note: standard notation uses X′ in place of X in many statistics books)

• Elements of β̂ measure relationships between the predictors and responses, in the context of all other predictor variables used in the model

• Values depend on the other predictors in the model, and differ in different models, paralleling changing interpretation of β_{i} parameters

• Center variables so that they have zero-mean, by subtracting sample mean (for each gene) before modelling. Good for numerical stability. One implication is that β̂_{0} = 0

♠ Collinearity of predictors

• Imagine a predictor x_{r} that is highly positively correlated with response y, so has a high positive regression coefficient estimate in the linear model using only that predictor. Fitting it in a more elaborate model with other xs changes things, often in unpredictable ways. The estimate β̂_{r} may be negligible, even negative. This will be experienced when the multiple predictors are correlated, and is due to other predictors dominating in explaining the response. This is called collinearity of predictors, and is the norm rather than the exception. At the other extreme predictors are orthogonal if XX′ is diagonal (so that they are uncorrelated), and in this case coefficient estimates do not depend on which other predictors are used in the model.

♠ Uncertainty and Significance of Predictors

• Standard inference results give standard errors (SDs) for each coefficient, say v_{j} for β̂_{j}, such that that symmetric error bars have the form of β̂_{j} ±c.v_{j} for appropriate constants c. This is a probability interval estimate of β_{j} (“confidence interval”). The regression function in matlab provided in this class plots 95% intervals.

• Similarly, standard inference gives ‘probability’ levels (or significance levels) for each coefficient. Predictors with non-significant parameters might be dropped from the model (see below).

♠ Simple stepwise fitting

• Choose a set of most highly correlated predictor variable and fit all of them.

• Look at the probability levels for each coefficient; if all are small (i.e., significant), stop, otherwise remove the predictor that is least significant, and repeat.

• Simple ‘backward selection’ procedure – an old, fairly crude method with some short-comings, but a standard method and a simple start on the difficult and pressing problem of selecting useful and relevant predictor variables in multiple regression models.

♠ Issues, concerns

• Many selection procedures: good properties, bad ones (e.g., backward/forward selection, AIC, BIC, etc)

• Purely data based: genes may be biologically relevant, in terms of networks, but ‘insignificant’ due to idiosyncacies of the data set

• Small sample issues: p should be ‘small’ relative to sample size n

• Over-fitting concerns: too many predictors, too few samples

• Wide interval estimates, wide prediction intervals in cases of small samples

• Bayesian methods of stochastic regularisation can improve predictions (see binary regression models later, where Bayesian methods are used)

♠ Networks

• Regression models help identify which genes are useful predictors of others, in terms of expression levels • Repeat with various genes selected as response variables

• One way of thinking – empirically – about network relationships (more refined methods would use Bayesian statistical methods - Bayes’ networks)

♠ Prediction

• Future response value y_{n+1} to be predicted at future predictors x_{n+1}

• New tumor sample, etc

• Predictive validity of regression model: How does the model stand up in out-of-sample prediction?

• Standard inference theory gives predictions in terms of – fitted/estimated/predicted value: ŷ_{n+1} = x′_{n+1}β̂ and corresponding standard error