Append to existing waves

When loading waves, we're given the option to overwrite existing waves with the same names. I'd like to see the additional option of appending waves, too.
I don't think this would be a good idea. If you load a wave whose name already exists as a wave, it's very likely that the same wave is already in your experiment. Why would one wish to append a wave to itself?

You can use concatenate to append a wave to another.
awirsing wrote:
If you load a wave whose name already exists as a wave, it's very likely that the same wave is already in your experiment. Why would one wish to append a wave to itself?
You can use concatenate to append a wave to another.


I'm talking about loading text files that have been generated by another application, and have headers that can be used as wave names.

I frequently need to load long time series of data, generated by other applications, that are broken up into multiple files, for example one per day over many days. When I load them into Igor I want them to be concatenated together as one set of waves. Using concatenate from the command line is not practical because there are many files and many waves per file, so I have to write a function, which may have to be changed for different data formats.

An "Append" option would allow me to do this in an instant, and seems a very logical addition, especially alongside the overwrite option.


If you load your data into separate data folders (one for each file), you can use AppendWaves given below to concatenate waves of the same name distributed over those data folders. Please look at the documetnation in the code, I hope it's clear enough. The code is tested, but not extensively. I wrote it in IP6.11/WinXP, but it might work with earlier versions.
Cheers, Wolfgang Harneit

// function AppendWaves(parentDF, DFpattern, killChildDFs, [sortedAppend, sortOptions, modelDF])
//
//  Given a data folder parentDF that contains wave1, Vwave and w3, AppendWaves appends
//  waves with the same name (i.e. wave1, Vwave and w3) found in any child data folder of parentDF,
//  if those childDFs match a name pattern DFpattern. If killChildDFs is true (non-zero), the childDFs
//  from which data were appended are killed. The sequence in which data folders are cycled (and thus
//  data are appended) can be influenced by setting sortedAppend=1 and sortOptions (see help for
//  SortList). If parentDF is empty initially, a modelDF may be specified to determine which waves are
//  to be extracted by AppendWave.
//
//  - if parentDF is an empty string, AppendWaves uses the current data folder
//  - if DFpattern is an empty string, AppendWaves uses the match-all pattern "*"
//  - if killChildDFs is non-zero, AppendWaves will kill child data folders with "exploited" data
//  - if sortedAppend is non-zero, AppendWaves will sort the child data folders before appending
//  - if sortedAppend is non-zero, sortOptions as used by SortList may be specified
//  - if modelDF is specified, AppendWaves uses it to determine which waves should be appended
//  - if modelDF is not specified, AppendWaves uses the parentDF as a model instead
//
//  Thus, AppendWaves("", "", 0) called with "root:" as the current data folder tries to append to all
//  waves currently living in "root:"; it collects waves from all top-level data folders and then kills those
//  data folders from which it collected waves. Use these "wildcards" with care...
//
//  The intended use is to load e.g. time series data spread over many files into separate folders called
//  "day1", "day2", etc. The folders can be put into a master folder called "alldays" so that we have a
//  folder structure like this:
//  >alldays
//  >>day1
//  >>--wave1 (0,1,2)
//  >>--wave2 (2,3,4)
//  >>day2
//  >>--wave1 (100, 101, 102)
//  >>--wave2 (202, 203, 204)
//  Now, call AppendWaves("alldays", "day*", 1) in order to collapse all runs into one:
//  >alldays
//  >--wave1 (0,1,2,100,101,102)
//  >--wave2 (2,3,4,202,203,204)
//
function AppendWaves(parentDF, DFpattern, killChildDFs, [sortedAppend, sortOptions, modelDF])
string parentDF, DFpattern
variable killChildDFs, sortedAppend, sortOptions
string modelDF
// get list of childDFs matching DFpattern
    if( strlen(parentDF) == 0 )
        parentDF = GetDataFolder(1)
    endif
    string childDFs = ListMatch( ChildDFList(parentDF), DFpattern+"*" )
    if( sortedAppend )      // default is 0 = false
        childDFs = SortList(childDFs, ";", sortOptions) // default is 0 = ascending case-sensitive
    endif                                           // alphabetic ASCII sort
    variable numChildren = ItemsInList(childDFs)        // count the "good" children
    if( numChildren == 0 )                      // ...none -- abort
        Abort "AppendWave found no child data folders"
    endif
// build WList from waves in modelDF or in parentDF
    string WList, saveDF = GetDataFolder(1)
    if( !ParamIsDefault(modelDF) )  // use modelDF if specified
        SetDataFolder modelDF
        WList = WaveList("*", ";", "")
    else
        SetDataFolder parentDF  // default to parentDF
        WList = WaveList("*", ";", "")
    endif
    variable numWaves = ItemsInList(WList)      // count waves to append to
    if( numWaves == 0 )                     // ...none -- abort
        Abort "AppendWave found no waves to append to in data folder \""+GetDataFolder(1)+"\""
    endif
   
    variable k, m, childDataAppended
    string childDF, currentWave, sourceWave, destWave
// cycle through childDFs
    for( k = 0; k < numChildren; k += 1 )
        childDF = StringFromList(k, childDFs)
        SetDataFolder childDF
        childDataAppended = 0   // don't want to kill childDFs that contained no data to append
// ... cycle through WList
        for( m = 0; m < numWaves; m += 1 )
            currentWave = StringFromList(m, WList)
            sourceWave = childDF+currentWave
            if( exists(sourceWave) == 1 )   // not all waves in child data folder may be relevant
                destWave = parentDF+currentWave
                if( exists(destWave) == 1 ) // parentDF may be empty initially
                    Concatenate/NP sourceWave+";", $destWave
                    print "C "+sourceWave+","+destWave
                else
                    Duplicate $sourceWave, $destWave
                    print "D "+sourceWave+","+destWave
                endif
                childDataAppended = 1
            endif
        endfor
        if( childDataAppended && killChildDFs )
            KillDataFolder childDF
        endif
    endfor
    SetDataFolder saveDF
end

// function/S ChildDFList(parentDF)
// returns a list of full paths to child data folders of data folder specified by parentDF
//  * if parentDF is an empty string, ChildDFList uses the current data folder
function/S ChildDFList(parentDF)
string parentDF
    if( strlen(parentDF) == 0 )
        parentDF = GetDataFolder(1)
    elseif( !DataFolderExists(parentDF) )
        Abort "ChildDFList cannot find specified data folder \""+parentDF+"\""
    endif
    parentDF = RemoveEnding(parentDF, ":") + ":"    // make sure parentDF ends in a colon
    variable k
    string CDFList = ""
    for( k = 0; k < CountObjects(parentDF,4); k += 1)
        CDFList += parentDF+GetIndexedObjName(parentDF, 4, k)+":;"
    endfor
    return CDFList
end
I second this.

It's not just useful in reading data in, but when analysing already current waves.

If I'm doing an analysis that counts things, I can append my results to the end of a new wave, without having to keep track of wave dimensions and the like.
masheroz wrote:
I second this.


Me, too!

masheroz wrote:

It's not just useful in reading data in, but when analysing already current waves.


Yes, and the use of Concatenate doesn't help if data is supposed to be appended to a wave that is already displayed somewhere. The use of Redimension or InsertPoints is a work-around but results in quite ugly code when it comes to placing DimLabels correctly.

May I suggest a Concatenate/A flag for appending data to the first wave of the list?
Here is an example of loading waves and appending to existing waves:
http://www.igorexchange.com/node/3788

Quote:
the use of Concatenate doesn't help if data is supposed to be appended to a wave that is already displayed somewhere


I don't understand that. You can concatenate whether or not a wave is displayed somewhere.
What I meant is that

Concatenate/O/DL/NP=0 {M_Initial,M_AppendThis}, M_out


has no effect on an existing graph on which M_initial (the wave I want append something to) is being displayed. However, I just realise that this is what I want:

Concatenate/O/DL/NP=0 {M_Initial,M_AppendThis}, M_tmp
Duplicate/O M_tmp, $NameOfWave(M_Initial)
KillWaves/Z M_tmp


Stupid me! Sorry, no need for an /A flag but a /FREE flag on Concatenate would still be appreciated ;-)
I don't understand why you need Duplicate and KillWaves. Perhaps that is because I am fuzzy on what you are trying to achieve.

Here is a example of contatenating additional columns to a display 2D matrix:
Make/O/N=(5,3) mat = p + 10*q
NewImage mat
Edit mat
Make/O/N=(5,2) newMat = p + 10*(q+3)
Concatenate/DL/NP=1/KILL {newMat}, mat


If you follow up, please explain what you are trying to do and give a complete example like the one above.
hrodstein wrote:
I don't understand why you need Duplicate and KillWaves. Perhaps that is because I am fuzzy on what you are trying to achieve.


Howard, I'm sorry for all the confusion. It turns out that I haven't read the documentation properly and I missed the following point:

"If destWave does exist and overwrite is not specified, the source waves' data is concatenated with the existing data in the destination wave."

Thanks for bringing this to my attention!