slowing down when processing large amount of data sets?

Share icon

Thu, 11/27/2008 - 09:31 am

Hello everyone,
I am currently using IGOR Pro for batch analysis. It involves extracting data from a directory of ascii files, performing some wave operations: wave copying/ box smoothing/ differentiation / wavestats/ Order(n) index-searching etc, and a few (line) curve fits. The main concern is, it seems to get drastically slower when I have a lot of "data sets", but I think I am only performing O(n) type calculations (i.e. traversing through each index of a wave).

To give an idea of what I mean by a "data set". It is a collection of 6-8 numeric waves that are of dimension 1000.
I collect information about each data set in list waves, i.e. containing constants and coefficients etc. I basically run through each data set by processing the list waves, and will go through the list 3 times.

I find that if I have a few 100s to about 1500 data sets, the time to completion of data processing is pretty much linear.
However, when I perform this on ~10 000 data sets it is not just 10x slower than performing on 1000. Maybe I have O(n2) or slower calculations that I ought to weed out, but is there something in Igor that would make my processing run slower just by increasing "data sets", eg am I letting it do too much virtual memory swapping?

Also, is the solution to this to perform Waveclears more often? Or, try to perform my procedures on a small batch at a time? (i.e. 10 000 data sets performed 1000 at a time with a lot of killwaves and waveclears in between?). I pretty much want to steer clear of multithread for now though,

Thanks a lot,

James
If you are accumulating lots of waves (thousands to tens of thousands) in one data folder (e.g., root) this can slow things down. The reason is that whenever Igor needs to find a wave that you reference, it has to look through a very long list of waves to find the right one. The solution is to use data folders to partition your waves.

You are probably already familiar with the data folder concept but if you need a refresher:

DisplayHelpTopic "Data Folders"

November 27, 2008 at 01:28 pm - Permalink