Statistics & Gene Expression Data Analysis Note 5: SVD & PCA

♠ Principal Components and Singular Factors

• p × n data matrix X (tall, skinny): rows are genes, columns are samples/microarrays

• Correlations, relationships, patterns among genes:

- clustering, similar expression patterns

- co-regulated genes, (up/down), genetic pathways

• Correlations, relationships, patterns among samples:

- different tumor types, clinical outcomes, cell cycle positions, tumor or normal, ...

• Correlation is a global measure: decomposes into constituent sources using - principal components analysis (PCA), or (equivalently)

- singular value decomposition (SVD) for singular factor analysis

♠ Motivating PCA and Factors: Two genes

• Two genes: gene 1, 2, sample j, expression levels x_{1,j}, x_{2,j} for j = 1, ..., n

• Imagine that there are numbers b_{1}, b_{2} and f_{1}, …, f_{n} such that

x_{1,j} ≈ b_{1}f_{j} and x_{2,j} ≈ b_{2}f_{j}

or, better,

x_{1,j} = b_{1}f_{j} + ϵ_{1,j} and x_{2,j} = b_{2}f_{j} + ϵ_{2,j}

for “small” ϵs and the ϵs are uncorrelated.

• f_{j} is the value of a factor on sample j, and the factor determines all of the correlation (relationship) between levels of expression on genes 1 and 2

• e.g., b_{1} = b_{2}, so x_{1,j} ≈ x_{2,j} or at least highly correlated

• e.g., b_{1} = 1, b_{2} = −1, so x_{1,j} ≈ −x_{2,j}, highly negatively correlated

• e.g., b_{1} = 0, so x_{1,j} and x_{1,j} are uncorrelated

• Linear regression format: the factor variable is a predictor of each x in the two linear regression models.

• Now suppose the ϵs are not so small, and are themselves related between gene 1 and 2

• Apply the same idea to the ϵs – introduces a second factor

• Then

x_{1,j} = b_{1,1}f_{1,j} + b_{1,2}f_{2,j} + ϵ_{1,j}^{∗} and x_{2,j} ≈ b_{2,1}f_{1,j} + b_{2,2}f_{2,j} + ϵ_{2,j}^{∗}

(relabelling b_{1} → b_{1,1}, b_{2} → b_{2,1} and f_{j} → f_{1,j})

• If the f_{1,j} are uncorrelated with the f_{2,j}, this describes patterns of dependence between xs ‘driven’ by the two separate, unrelated factors

• Linear regression format: the 2 factor variables are predictors of each x in the two linear regression models.

♠ PCA and Factor Decompositions

• p × n data matrix X – p genes, n samples

• Take row (gene) i and sample (microarray) j

• Singular value decomposition (SVD) of X can be expressed exactly as

x_{i,j} = b_{i,1}f_{1,j} + b_{i,2}f_{2,j} + ⋯ + b_{i,n−1}f_{n−1,j} + b_{i,n}f_{n,j}

for some numbers b_{···} and f_{·}. (this is just linear algebra; no statistics, and no magic).

• Generally, higher order b_{i,n} terms are small, so

x_{i,j} = b_{i,1}f_{1,j} + b_{i,2}f_{2,j} + ⋯ + b_{i,k}f_{k,j} + ϵ_{i,j}

for some k < n and some ‘small’ terms ϵ_{i,j} that are uncorrelated across genes i and arrays j (i.e., they are small, residual ‘noise’ terms)

• Linear regression format: the k factor variables are predictors of each of the p x response variables in p separate, parallel linear regression models.

• The k factor variables explain variability in the expression patterns of the many genes and represent k different aspects of the correlations, structure, patterns exhibited among the genes and across samples

• The regression parameters b_{i,r} for gene i represent different weightings, or loadings, on factor r for this gene – the factors influence/explain the variation in genes differently due to differing values of these loadings.

• The factors have various alternative names: principal components, principal factors, singular factors, among others

♠ PCA and Factor Decompositions: Matrix/vector form

• Sample j = 1, …, n, with x_{j} = column j of X

x_{j} = b_{1}f_{1,j} + b_{2}f_{2,j} + ⋯ + b_{n}f_{n,j}

or

x_{j} = ∑_{r=1}^{n} b_{r}f_{r,j}

where each b_{r} is a p × 1 column vector of the loadings for all genes on factor r as r = 1, …, n

• Or,

x_{j} = Bf_{j}

with B = [b_{1}, …, b_{n}] and

f_{j} = ( f_{1,j} ; f_{2,j} ; ⋮ ; f_{n,j} )

• Or, X = BF

where now F = [f_{1}, f_{2}, …, f_{n}]

♠ Important Mathematical features

• The factor variables are uncorrelated, so represent different underlying sources of covariability in the X data

• Sample correlations between any two factor variables is zero

• Sample variance of each factor variable is 1

• Formally, F is an orthogonal matrix: F′F = I and FF′ = I, where I is the n × n identity matrix •

X = BF = ADF

with - n × n diagonal D = diag(d_{1}, …, d_{n}) of non-negative values in decreasing order

- the singular values of X

- A is p × n matrix such that B = AD

- A has orthonormal columns: A′A = I the p × p identity matrix

• In terms of A, D,

x_{j} = ADf_{j}

or

x_{j} = ∑_{r=1}^{n} a_{r}d_{r}f_{r,j}

or

x_{i,j} = a_{i,1}d_{1}f_{1,j} + a_{i,2}d_{2}f_{2,j} + ⋯ + a_{i,n}d_{n}f_{n,j}

• Singular values describe relative importance of factors in describing relationships and variability in data matrix

• Percent “total variation explained” by factor j is 100d_{j}^{2}/ ∑_{i=1}^{n} d_{i}^{2}

• Elements in a_{r} – column r of A – describe relationships among genes due to factor r. A is called the factor loadings matrix

• Elements in rows r of F describe relationships among samples/microarrays due to factor r

♠ Properties and More Interpretation

• The factors (principal components) are themselves linear combinations of the data variables, namely f_{j} = D^{−1}A′x_{j} for each sample j, or

f_{i,j} = a_{1,i}d_{1}^{−1}x_{1,j} + a_{2,i}d_{2}^{−1}x_{2,j} + ⋯ + a_{p,i}d_{n}^{−1}x_{p,j}

• In fact, among all possible (unit length) linear combinations of the data variables, the factors are those

that explain the most “variability” in the x data, in the sense that

– The first factor is the linear combination of the data that has the largest sample variance. (For any vector c, such that c′c = 1, compute the n values c′x_{j}, (j = 1, …, n), and then find the sample variance of these n values; choose another vector c, do it again; the largest variance arises when c is the first column of AD^{−1} so that the linear combination is the first factor.)

– The second factor is the linear combination of the data that has the the largest sample variance once corrected for the first factor (and subject to being orthogonal)

– Many other properties (e.g., see Ripley section 9.1)

• Relationship to eigen-decomposition of sample variance matrix S. Take (with no loss of generality) centred data (sample mean subtracted by rows) so that (n − 1)S = XX′. SVD implies

(n − 1)S = AD^{2}A′

• A is matrix whose columns are eigenvectors of S

• Squared singular values are eigenvalues of S (up to a constant)

• “Variation explained” is sum of squares of eigenvalues

• p ≥ n : - at most n non-zero eigenvalues: S is rank deficient, singular (our case with p >> n – many more genes than samples, typically)

• (n − 1)S = ∑_{j} d_{j}^{2}a_{j}a′_{j}

• Variance decomposition: for gene i, s_{i} = ∑_{j} a_{i,j}^{2}d_{j}^{2} showing the role of the factor loadings a_{i,j} of gene i on factor j in explaining variance

• Covariance decomposition: for genes i, k, s_{i,k} = ∑_{j} a_{i,j}a_{k,j}d_{j}^{2} showing the similar role of the factor loadings, and their signs

♠ Plotting Data and Factors

• Often informative displays are achieved by plotting factors against sample number and scatter plotting data on pairs of factors

• Useful for discrimination of samples: finding patterns and structure in the n samples that may be related to a biological state or features (e.g., tumor versus normals)

• First factor often represents average levels of genes in each sample (unless data are first centred)

• Clustering methods can be applied, often most usefully, to factors rather than the full data set – computational efficiencies

• Higher order factors can represent small, idiosyncratic features in data

• Factors can be most useful in regression models as predictors of outcomes

• Exploration of columns of A to identify factor loadings patterns: genes “related” in terms of sign and magnitude on a given factor

• Use of image (heat map) displays

• Some matlab functions (ShowGene, pairs and bcpairs, scatter3 and sc3, imagesc, imagecf, ...)

♠ Practical consideration

• PCA/SVD depends on scale of measurement of variables

• Gene expression on a standard scale – same for all genes – best on some kind of log scale

– require normalisation off all arrays to a standard scale

• Exploratory uses on selected subsets of genes (variables)