Exporting/appending summary data

Share icon

Sun, 05/22/2016 - 07:21 am

I'm analyzing fairly large experiments, each one with its own .pxp file. At the end of my analysis of each one, I intend to export data in the form of a few characteristic calculations to a "master file" that includes just the summary data-- a handful of values and possibly descriptions from each experiment.

The idea is that I could then analyze the trends of these summary data in another .pxp file. This "master" .pxp file would have just the summary data, and would exclude all of the raw data from each individual experimental .pxp that generated the summary data.

As I think about how to accomplish this, the best way I've been able to come up with is to create an external .itx file for each wave of summary data that I want to maintain. When I want to add new data upon completing analysis of a new experiment, I could load the wave using the "LoadWave" command, concatenate the new summary data, and then use the "Save" command to update the wave with the newly analyzed data.

I could then re-load this data into the master .pxp file each time I want to do updated analysis on it.

Is this the best way to do this, or is there a more elegant way to populate information in the "master" experiment file?

Thanks for any advice or suggestions.
Quote:
As I think about how to accomplish this, the best way I've been able to come up with is to create an external .itx file for each wave of summary data that I want to maintain.


This is a reasonable approach. However, I think I would use Igor binary files instead.

You can save a wave to an Igor binary file use the Save operation and load it using the LoadWave operation. You should make sure that you are "sharing" the wave file. This means don't use /C or /H with the Save operation and don't use /H with the LoadWave operation. Omitting these flags means that the persistent wave data resides in the file and is not copied into any experiment.

Using Igor binary files allows you to access waves that are part of an unpacked experiment. (See the "Saving as an Unpacked Experiment File" help topic for details.) This means you can have a "summary experiment", the waves of which any experiment can access. The summary experiment could include graphs, tables and procedures for examining the summary waves.

May 22, 2016 at 11:24 am - Permalink

Hi Jeremy,

I am doing something similar. My solution is a new master pxp which reads in the data from a directory of pxps, and then summarises them. The loading of data goes into a data folder for each experiment to keep everything tidy. I also delete waves that are not needed. You can write something that will grab a certain wave from each folder (they will all be the same) and concatenate them with /NP to just append the data?

May 22, 2016 at 11:28 am - Permalink

I use a dedicated data folder in my experiments to store the "public" data. The summarizing routine checks for this DF and loads (LoadData) it into a new DF for each experiment (that keeps the data together). Appending data to waves is something I would do in a second step (to keep the original data together - it might be useful for checkups).
HJ

May 22, 2016 at 12:37 pm - Permalink

Thanks to everyone for the helpful suggestions!

I like the idea of saving the summary data in a separate folder in the original experiments and then loading that folder into the master experiment. I think I need to familiarize myself more with the LoadData command.

May 23, 2016 at 06:28 am - Permalink

FWIW, when your intent is to export data from a graph in an experiment across a set of experiments in order to create a new "overview" experiment of just the selected graphs, you might be interested in the SnapIt! package.

http://www.igorexchange.com/project/SnapIt

This takes the approach to use the Save Graph and Merge Experiment methods.

--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAH

May 23, 2016 at 11:33 am - Permalink