Data folders, dependencies, and variables

Share icon

Tue, 08/26/2008 - 07:57 am

I generate many data sets, where each data set consists of several waves of data. From data set to data set, each wave contains the same physical quantity (i.e. wave0 of the data set is always average optical power, wave1 is always the RF power, etc). Obviously different experimental runs have different numbers within each wave.

This appears to be exactly the sort of situation in which the data folders might be useful. Within the root data folder, I can create sub-folders for each experimental run (named by say date and time). The idea is that by then changing the "current" data folder, I could quickly see all my plots and analysis for different experimental runs.

The difficulty is that I need to perform various mathematical scalings and manipulations on my data in order to plot, analyze, etc. (using both waves and single values as the input to the formulas). It would be messy and inefficient to create these calculated waves in each sub-folder. The mess and inefficiency is due to the fact that there is just one formula for a given calculated wave (say avg optical power converted from A/D volts to dBm) that gets repeated over and over.

It seems like it would be far more efficient (and definitely cleaner in terms of data storage and management) to be able to have a single instance of each calculated wave live in the "root" directory of the data folder structure. Then the calculated wave is based on the waves that are in the "current" sub-folder.

Creating simply dependencies does not work. If anyone has some tips or tricks, please let me know.
Thanks in advance.
Andy
acfunk2 wrote:
I generate many data sets, where each data set consists of several waves of data. ...


IMO, based on your description, storing your waves in data folders will in the long term be a better administrative strategy. Creating a function to work on a wave of a particular name existing across multiple data folders may take an upfront investment in time, however the payoff is knowing over the long term that each data folder belong to some specific event marker (date, time, location ...).

You could also create a master function that duplicates waves from a given folder into a "master" folder and use dependencies to keep the calculations "live". However, once you create a function to duplicate FolderX(waves) -> Master(waves), running an update function to redo all the required calculations on Master(waves) avoids the problems with dependencies. The overall approach is then ...

Function DoItAll()

     givenFolder = SelectFolder()
     DuplicateToMaster(givenFolder)
     UpdateMaster()
     return 0
end


Running this via a Control Panel with associated popups and buttons could be a subsequent step.

--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAH

August 26, 2008 at 01:36 pm - Permalink

I'm not a fan of dependencies because you can create contraptions that are difficult to understand and debug. However, you can create a dependency across data folders. Here is an example:

NewDataFolder TestFolder
Make/O root:TestFolder:TestWave = p
Make root:dependentWave := root:TestFolder:TestWave
Display dependentWave
root:TestFolder:TestWave = sin(x/8)


To programmatically create a dependency from a dynamically chosen wave:
Function Test()
    Wave dependentWave = root:dependentWave
    Wave masterWave = :TestWave     // In the current data folder
    String formula = GetWavesDataFolder(masterWave,2)
    SetFormula dependentWave, formula
End

August 26, 2008 at 06:46 pm - Permalink