Statistics & Gene Expression Data Analysis Note 7: Bayesian Regression

♠ Conjugate Priors on Regression Parameters

• Model in matrix form:

y = X′β + ϵ

• Use notation H = X′ for comparability with traditional statistics notation: H is n × p design matrix

y = Hβ + ϵ, ϵ ∼ N(0, σ^{2}I) = N(0, ϕ^{−1}I)

♠ LSE formulæ:

•

β̂ = VH′y and V^{−1} = H′H

♠ (Simple) Standard Shrinkage Priors

• Conjugate normal prior centred at zero: shrinkage towards zero

• Genes centred: zero is canonical “null” hypothesis of no (partial) association

• Scale: scale of gene expression, standardised scales

β|ϕ ∼ N(0, τϕ^{−1}I)

• Intercept: include in β - different status: different scale?

β|ϕ ∼ N(0, ϕ^{−1}C^{−1})

• Prior precison matrix C (up to constant ϕ) is diagonal – common value τ^{−1} for genes, possibly different (larger) for intercept

• Prior on residual variance/precision: ϕ ∼ Ga(a/2, b/2) with prior variance estimate b/a and degree of fredom a. Vague reference prior is special case a, b → 0

♠ Features of Posterior Under Shrinkage Prior

• β|ϕ, y ∼ N(b, ϕ^{−1}B^{−1})

• B = C + H′H and b = B^{−1}H′y

• Special limiting cases: vague prior (zero precision) - B = H′H and b = β̂

• Otherwise, shrinkage towards zero induced by prior: ||b|| < ||β̂||

• Decision Theory: Bayes’s estimates improve expected performance in estimating β - quadratic (or other convex) loss functions

- Bayesian and frequentist measures of risk in estimation • In practical terms, shrinkage generally stabilises/regularises estimation and improves robustness in

predictions

• Large p questions: LSE/likelihood methods inapplicable – need for regularisation

• Log posterior (conditional on ϕ) = log likelihood + log prior: a constant plus a term proportional to

(y − Hβ)′(y − Hβ) + β′Cβ

• Posterior mean/mode b represents shrunken/regularised MLE (=LSE)

♠ Marginal Posteriors for b and ϕ

• ϕ|y ∼ Ga(a′/2, b′/2) with a′ = a + n and b′ = b + q where q > 0 can be written in one of several useful

forms: - simplest for computation: q = y′e where e = y − Hb is the estimated residual vector

- quadratic form representation: q = y′P′y where P = I − HB^{−1}H′

- interesting limiting case of vague reference prior: we know b → β̂ and now also q → usual residual

sum of squares in LSE regression with P → H(H′H)^{−1}H′

• Common point estimate of σ^{2} = ϕ^{−1} is reciprocal of posterior mean of ϕ, namely s = b′/a′

• β|y ∼ T_{a′}(b, sB^{−1})

- credible intervals for elements of β are derived from Student T distribution on a′ degrees of freedom • (see matlab code: mregbayes)

• Role of C, Key example:

- C_{1,1} =large (intercept term, vague prior)

- C_{i,i} = τ^{−1} otherwise – common precision for (common scale) genes

- Marginal likelihood function for τ (see matlab code, examples) to assess τ. Can be shown that marginal likelihood p(y|τ) implies

log(p(y|τ)) = constant + 0.5 log(|C|/|B|) − 0.5n log(y′e)

♠ Shrinkage Priors More Generally: Considerations

• Different degrees of shrinkage across variables (genes) via different diagonal elements in C

• General question of learning from data about shrinkage parameters: inference on C generally

• Orthogonal designs: e.g., Factor regressions and other models: design is orthogonal (by design!) - H′H is diagonal

- so that B is diagonal when C is

- elements of β are uncorrelated under the posterior as well as the prior (dependent only through scale ϕ)