# maximum matrix sizes for MatrixSVD and PCA

ChrLie

Wed, 01/07/2015 - 07:41 am

I was wondering if there is a clear restriction in the matrix sizes, e.g.

Make/O/N=(1000,1000) M_in=trunc(abs(enoise(100)))

MatrixSVD M_in

PCA/SCMT M_in

MatrixSVD M_in

PCA/SCMT M_in

would execute, whereas as matrix with the same number of points but n(rows) >> n(cols), such as

Make/O/N=(100000,10) M_in=trunc(abs(enoise(100)))

would produce "out of memory" errors. Are there ways to avoid this (on Mac OS)?

PCA/SCMT M_in

are not equivalent because the PCA operation still prepares an input "D" matrix for you unless you specify the /U flag.

If you are interested in PCA you might also be interested in IP7 built-in ICA.

A.G.

WaveMetrics, Inc.

January 7, 2015 at 10:10 am - Permalink

Yes, I'm aware of that. Initially I tried PCA "by hand" to better understand Igor's PCA output. This is when I found that some matrices were too large for MatrixSVD - those sizes also failed on PCA - I guess because it uses the same SVD code.

I am VERY interested in IP7 ;-)

Do you have any recommendation if my matrices for PCA would as large as above?

EDIT:

it seems that the /LEIV flag avoids out-of-memory issues, although I'm not sure (yet) about the consequences...

January 8, 2015 at 02:12 am - Permalink

In that case I recommend that you contact support for information about the technology preview program.

The problem, as you know, is that the covariance matrix gets to be too large for 32 bit address space. In some situations you do not need to work with the covariance matrix. Use MatrixOP to condition your input and then use /U.

This flag helps but only AFTER the operation creates the covariance matrix. In most situations there are less components than there are rows in your data so there is no sense to spend time computing eigenvalues corresponding to noise. This is where it makes sense to limit the eigenvalues range with the /LIEV flag. This reminds me that in IP7 you have a new option to compute partial svd.

A.G.

WaveMetrics, Inc.

January 8, 2015 at 09:51 am - Permalink