Loading a whole bunch of files

Share icon

Thu, 03/02/2017 - 08:24 am

Dear igorians,

I am trying to load a lot of files and obviously this takes a lot of memory. I tried sharing the files with the current experiment (as compared to copying) and I still stuck.

I can write a short script displaying the files one by one (as I have to sift through all this data one by one) and go around the issue.

Any other ideas as to what can be done here?

best,
_sk
Out of curiosity how large are your files? As I understand it Igor Pro can handle 2-4 GB, which seems like a lot to me. If you really are hitting the memory ceiling I guess you have to load your waves in smaller batches, view them, delete them and load the next batch. Maybe you could force your waves to be single precision, or where appropriate integer, instead of double precision? If you have a lot of small waves, maybe you are having memory fragmentation issues. In that case you might want to combine waves, but I'm no expert.

Or load the files on-demand. Use the Open /MULT=1 to get the list of files you want to work on, but don't actually load the file until it is needed and kill it before the next file is loaded. It all depends on what you are doing with your data.

March 2, 2017 at 09:15 am - Permalink

It's not clear what your actual problem is. Are you getting out of memory errors? Some other error? Performance problems?

If you're running out of memory, you could try using a 64-bit version of Igor. If you're still using IP6, and you're on Windows, you could try Igor Pro 6 64-bit (http://www.wavemetrics.net/Updaters/ReadMe%20(64-bit).html). If you're using Igor 7, it ships with 32-bit and 64-bit versions on both platforms. Just run the 64-bit version and that may solve your problem.

March 2, 2017 at 09:20 am - Permalink

Is your machine in principle capable of having all the files in memory? If yes Igor Pro 7 64bit is the solution. I don't recall its memory limits, but it should be much more of what a typical computer can have.
If not, you need to think about the data types. Do you need floating point data (maybe real is enough) or can you use integer waves? Maybe you can even use integer waves with 16bit?

March 2, 2017 at 09:24 am - Permalink

Thank you all for posting suggestions.

The particular situation I was faced with was loading multiple small files beyond the amount of physical RAM.

Luckily I don't need all the files to be loaded at the same time, so I will just swap out to and from disk unneeded files and fit within RAM.

best,
_sk

March 3, 2017 at 05:02 am - Permalink