multiple data(txt file) import

Could someone explain how to import multiple data files? I just started to learn Igor instead of Origin so I think I need some time to get used to this. Tutorials were useful but it doesn't solve my question. I have bunch of data files but load them one by one currently...I've searched forum (and also Igor manual) and found some topic about 'Displaying a Multi-Selection Open File Dialog' but wasn't really helpful yet. Shoud I write a procedure file to do that? or any simple way I missed?
I recently wrote some code to do exactly this.
I've trimmed out all of the stuff that is not important to you.
You will need to use the multiopenfiles XOP (the one you have been looking at already), pop it in the IgorExtensions folder as instructed in the installation file. Once there, the code below will work.
This will open each file, and store the filename in a wave called 'Results'.
Getting the LoadWave bit right for your text file files can be sticky sometime (I had to ask for help here).

I hope this is enough to get you going.
Jason.

#pragma rtGlobals=1     // Use modern global access method.
Function MultiLoad()
multiopenfiles
string text_string = S_filename
Make/O/T/N=(ItemsInList(text_string, ";")) text_wave
text_wave = StringFromList(p, text_string, ";")
Make/O/T/N=(numpnts(text_wave),1) Results
For (nof=0;nof<=numpnts(text_wave)-1;nof+=1)
LoadWave/A/D/J/W/K=0/V={" "," $",0,0}/L={0,2,0,0,0}  text_wave[nof]
    Results[nof][1]=S_filename
    KillWaves/Z wave1
endfor
End Function


Thanks, Jason.
This is exactly what I wanted.
Now I feel like getting some ideas on Igor...
If you're using Igor 6.1 or greater, we don't recommend using the XOP to do this. It's not that there is a problem with the XOP, but Igor 6.1 and greater has this functionality built in, and so it will simplify your life if you use the built in functionality.

Here is an example that's almost directly copied from the Igor help:

Function/S DoLoadMultipleFiles()
    Variable refNum
    String message = "Select one or more files"
    String outputPaths
    String fileFilters = "Data Files (*.txt,*.dat,*.csv):.txt,.dat,.csv;"
    fileFilters += "All Files:.*;"

    Open /D /R /MULT=1 /F=fileFilters /M=message refNum
    outputPaths = S_fileName
   
    if (strlen(outputPaths) == 0)
        Print "Cancelled"
    else
        Variable numFilesSelected = ItemsInList(outputPaths, "\r")
        Variable i
        for(i=0; i<numFilesSelected; i+=1)
            String path = StringFromList(i, outputPaths, "\r")
            Printf "%d: %s\r", i, path
            // Add commands here to load the actual waves.  An example command
            // is included below but you will need to modify it depending on how
            // the data you are loading is organized.
            //LoadWave/A/D/J/W/K=0/V={" "," $",0,0}/L={0,2,0,0,0} path
        endfor
    endif
   
    return outputPaths      // Will be empty if user canceled
End


You will have to work out a LoadWave command that works for the type of data you are trying to load. You should carefully read the command help for the LoadWave operation. I also recommend reading the help for how to load waves. You can access that by executing the following command in Igor:
DisplayHelpTopic "Loading Waves"


If you need additional help figuring out how to load your data, feel free to send an email to support@wavemetrics.com describing how you want the data to be imported into Igor and also send a sample data file. Or you can post those things here if you'd prefer.
For the relevant help, execute this in Igor:
DisplayHelpTopic "Loading Waves Using Igor Procedures"

I tried all the codes here and feel much better.
Now I can import several data files and things much easier. As you guys recommended, I read help topics and tried another code such as 'Loading All of the Files in a Folder', etc..BTW, I got an error message while compiling the sample code and couldn't figure out what was wrong. It says,
"unknown/inappropriate name or symbol

Error in Procedure:LoadAndGraphAll
result=LoadAndGraph(fileName,pathName"
with 'LoadAndGraph' highlighted.

Here's my problem. Unfortunately, I'm not 100% understanding all the codes. but for now just wanna try to combine and modify sample codes to make my work easier.

just one more quick question. To load multiple files, sample code from aclight works fine but I have more than 20 files in the folder so when the waves automatically named it's really hard to tell them. is it possible to name all the waves from file name autometically? If this issue also included in the help file let me know. I'll search them again.. Thanks again~

I kinda like plot quality and the way Igor handling plots...but the troubles I mentioned here are relatively difficult and sometimes annoying to me than Origin...but I'm keep trying to learn cause Igor definitely has some advantages for my work if I get used to it.
>I got an error message while compiling the sample code

The LoadAndGraphAll function calls the LoadAndGraph function so you need to copy and paste both functions from the help into the procedure window.

>is it possible to name all the waves from file name autometically?

Yes. You can use the file name as an input to the LoadWave /A flag which sets the base name for the waves created by LoadWave. For example:

String baseName = CleanupName(fileName,0)
LoadWave /A=$baseName


Alternatively you can use the LoadWave /B flag but this gets complicated.

It is easier for me to write some starter code for you than to explain the details here. If you want me to do that, create a zip archive of some sample files and send them to support@wavemetrics.com along with an explanation of what you want to do with the columns and how you want to name them. If it is OK with you, I will write some starter code and post it on IgorExchange as an example.
hrodstein wrote:

Alternatively you can use the LoadWave /B flag but this gets complicated.

It is easier for me to write some starter code for you than to explain the details here. If you want me to do that, create a zip archive of some sample files and send them to support@wavemetrics.com along with an explanation of what you want to do with the columns and how you want to name them. If it is OK with you, I will write some starter code and post it on IgorExchange as an example.


I have a slightly different problem - I am trying to read a file that looks something like below:

1 5
1 0 0 0 0
2 0.1 0.1 0.1 0.1
3 0.2 0.2 0.2 0.2
4 0.3 0.3 0.3 0.3
5 0.4 0.4 0.4 0.4
2 5
1 0 0 0 0
2 0.1 0.1 0.1 0.1
3 0.2 0.2 0.2 0.2
4 0.3 0.3 0.3 0.3
5 0.4 0.4 0.4 0.4
3 5
1 0 0 0 0
2 0.1 0.1 0.1 0.1
3 0.2 0.2 0.2 0.2
4 0.3 0.3 0.3 0.3
5 0.4 0.4 0.4 0.4

So I have blocks of data in 5 columns separated by a two number row which correspond to a run ID and total number of rows in the subsequent block.

In each of these blocks, first column is my x-variable and columns 2 through 5 are the y-variables. Is there a way for me to write a program to read each block separately and append the wave names of the five columns with corresponding run ID like C1_1, C2_1, C3_1, C4_1, C5_1 for the columns in the first block where as C1_2, C2_2, C3_2, C4_2, C5_2 for columns in second block? I am trying to play with /B flag of LoadWave but I didn't know how to make the N flag of columnInfoStr (where columnInfoStr is input to /B just as in Igor Help) to read a variable.
I would probably call LoadWave/J to load a single line containing the two numbers and then call it again to load the 5x5 block of numbers, then repeat till done.

Another approach is to use Open/FReadLine/Close to read each line one at a time and to use sscanf to convert the line into numbers, then store the numbers in waves.

If you want help, send a zipped sample file to support@wavemetrics.com with a description of what you want to do with the data, what OS you are running and what version of Igor.

By "what you want to do with the data" I mean, do you want to load each 5x5 block into a 5x5 wave? Do you want to into 5 separate 1D waves? Do you want to contatenate all of the blocks into one wave? And what do you want to do with the run IDs? And how do you want to name the waves?

I am having trouble reading the wave names from the file name. When using:

String baseName = CleanupName(fileName,0)
LoadWave /A=$baseName

with the Open command, I get an error (wave name too long) because fileName returns the entire path to the file, not the file name only. Any way to deal with that?

Thanks!

Quote:
I get an error (wave name too long) because fileName returns the entire path to the file, not the file name only. Any way to deal with that?
Thanks!


Function/S FilePathToWaveName(path)
    String path     // Assumed to use colon separators
   
    // Get last element of path without extension
    String name = ParseFilePath(3, path, ":", 0, 0) // e.g., MyFile.txt
   
    name = CleanupName(name, 0)

    return name
End

hello guys, i have a slightly different problem..i have many text files each containing 2 columns..i have to load only the 2nd column from each file and assign the wave name with corresponding filename...

what i did was to put all files in a separate folder..create a text wave containing name of each file in each row (use "multiopenfiles" to get the string containing all the fullpaths/wavenames..then by writing a simple code i made the text wave which now contains one item in each row..since they are full paths, i replaced the unnecessary parts of the strings)...then i used the '"load all waves in a folder" with autonames and constructed a matrix.. then excute another code to construct another matrix only with the alternate columns (since the 1st column in each wave is not required) and extract them as 1d waves but with names from the text wave to each of the 1d waves in the same order..

i suppose there should be some smarter way..any advice ??

thanks in advance.. :)
I don't follow what you have done entirely but . . .

You can get a list of file names in a folder using IndexedFile.

You can specify the name to be used for a wave loaded with the LoadWave operation using the /B flag. You can also specify that a column is to be skipped using /B.
aclight wrote:

Function/S DoLoadMultipleFiles()
    Variable refNum
    String message = "Select one or more files"
    String outputPaths
    String fileFilters = "Data Files (*.txt,*.dat,*.csv):.txt,.dat,.csv;"
    fileFilters += "All Files:.*;"

    Open /D /R /MULT=1 /F=fileFilters /M=message refNum
    outputPaths = S_fileName
   
    if (strlen(outputPaths) == 0)
        Print "Cancelled"
    else
        Variable numFilesSelected = ItemsInList(outputPaths, "\r")
        Variable i
        for(i=0; i<numFilesSelected; i+=1)
            String path = StringFromList(i, outputPaths, "\r")
            Printf "%d: %s\r", i, path
            // Add commands here to load the actual waves.  An example command
            // is included below but you will need to modify it depending on how
            // the data you are loading is organized.
            //LoadWave/A/D/J/W/K=0/V={" "," $",0,0}/L={0,2,0,0,0} path
        endfor
    endif
   
    return outputPaths      // Will be empty if user canceled
End


I am using this to load multiple runs of the same data. I successfully have that done but I have to change the names of the waves for each file. I would ideally like to append just the file name to each column or I think just using data folders labeled with the file name would be sufficient if my understanding of them is correct. I know how to create the data folders but not when it is using this path string for each file so I am curious to see how this is done. Thanks guys.
If you want to control the names of the waves use the /B flag.

If you want to create a data folder for each file, call NewDataFolder/S before loading the file and SetDataFolder :: after.
I am a new user of Igor. I have problem with loading some data and I would appreciate if someone can help me.
I have a single data file including two columns and numbers of rows, in which any 1012 rows should be loaded in separated data folders. Does anyone have a code for this problem?
Please create a new topic for this question.

Before doing that, please execute the following commands and read the relevant help:
DisplayHelpTopic "Loading Waves"
DisplayHelpTopic "Loading Waves Using Igor Procedures"


Quote:
I have a single data file including two columns and numbers of rows, in which any 1012 rows should be loaded in separated data folders. Does anyone have a code for this problem?


I don't understand the question so, in the new topic, please rephrase it with more detail, and attach a zipped archive of your data file.
aclight wrote:
If you're using Igor 6.1 or greater, we don't recommend using the XOP to do this. It's not that there is a problem with the XOP, but Igor 6.1 and greater has this functionality built in, and so it will simplify your life if you use the built in functionality.


SO where is the built in functionality?
Ki2 wrote:
aclight wrote:
If you're using Igor 6.1 or greater, we don't recommend using the XOP to do this. It's not that there is a problem with the XOP, but Igor 6.1 and greater has this functionality built in, and so it will simplify your life if you use the built in functionality.


SO where is the built in functionality?


I think Adam was referring to the /MULT option to the builtin Open command, as in the example code he posted.

--Jim Prouty
Software Engineer, WaveMetrics, Inc.
Hi Adam,

I have been using igor so far primarily for plotting and fitting using dialogs. Now I feel a need to write my own procedures for various purposes. One of them is to import multiple .csv or .dat files as general text (as the files have some header information which I want to skip). I was trying to use this code for that purpose but I am not able to figure out a few things - what are the different kind of flags and which one to use for general text import so that I also have an option of skipping some of the waves while importing. When I use your code above to import such files (I included the LoadWave line as it is) is was recognizing the files as delimited text. Please see the attached file as an example.

Thanks
aclight wrote:
If you're using Igor 6.1 or greater, we don't recommend using the XOP to do this. It's not that there is a problem with the XOP, but Igor 6.1 and greater has this functionality built in, and so it will simplify your life if you use the built in functionality.

Here is an example that's almost directly copied from the Igor help:

Function/S DoLoadMultipleFiles()
    Variable refNum
    String message = "Select one or more files"
    String outputPaths
    String fileFilters = "Data Files (*.txt,*.dat,*.csv):.txt,.dat,.csv;"
    fileFilters += "All Files:.*;"

    Open /D /R /MULT=1 /F=fileFilters /M=message refNum
    outputPaths = S_fileName
   
    if (strlen(outputPaths) == 0)
        Print "Cancelled"
    else
        Variable numFilesSelected = ItemsInList(outputPaths, "\r")
        Variable i
        for(i=0; i<numFilesSelected; i+=1)
            String path = StringFromList(i, outputPaths, "\r")
            Printf "%d: %s\r", i, path
            // Add commands here to load the actual waves.  An example command
            // is included below but you will need to modify it depending on how
            // the data you are loading is organized.
            //LoadWave/A/D/J/W/K=0/V={" "," $",0,0}/L={0,2,0,0,0} path
        endfor
    endif
   
    return outputPaths      // Will be empty if user canceled
End


You will have to work out a LoadWave command that works for the type of data you are trying to load. You should carefully read the command help for the LoadWave operation. I also recommend reading the help for how to load waves. You can access that by executing the following command in Igor:
DisplayHelpTopic "Loading Waves"


If you need additional help figuring out how to load your data, feel free to send an email to support@wavemetrics.com describing how you want the data to be imported into Igor and also send a sample data file. Or you can post those things here if you'd prefer.

example.csv
Hi,

Sorry to hijack the thread. I'm using this to load multiple CSVs

Function/S DoLoadMultipleFiles()
    Variable refNum
    String message = "Select one or more files"
    String outputPaths
    String fileFilters = "Data Files (*.txt,*.dat,*.csv):.txt,.dat,.csv;"
    fileFilters += "All Files:.*;"

    Open /D /R /MULT=1 /F=fileFilters /M=message refNum
    outputPaths = S_fileName
   
    if (strlen(outputPaths) == 0)
        Print "Cancelled"
    else
        Variable numFilesSelected = ItemsInList(outputPaths, "\r")
        Variable i
        for(i=0; i<numFilesSelected; i+=1)
            String path = StringFromList(i, outputPaths, "\r")
            Printf "%d: %s\r", i, path
            // Add commands here to load the actual waves.  An example command
            // is included below but you will need to modify it depending on how
            // the data you are loading is organized.
            //LoadWave/A/D/J/W/K=0/V={" "," $",0,0}/L={0,2,0,0,0} path
        endfor
    endif
   
    return outputPaths      // Will be empty if user canceled
End


I was wondering if anyone can help with my LoadWave command. The CSVs have two columns: "Time" & "Voltage". I want the output waves to be "CSVfileName"Time and "CSVfileName"Voltage but I can't figure it out.

Help is much appreciated! Thanks,

Matt
vince_moaloka wrote:
I was wondering if anyone can help with my LoadWave command. The CSVs have two columns: "Time" & "Voltage". I want the output waves to be "CSVfileName"Time and "CSVfileName"Voltage but I can't figure it out.


I haven't figured out a straightforward way to do this, but it can be done using the /M and /B flags of LoadWave. You need to create Igor-friendly strings for the wave names and condense those into a format that the /B flag will like. I've only gotten it to work by using the /M flag and loading each column separately as a "matrix" then redimensioning that back to a regular wave. Using this approach you need a separate "LoadWave" command for each column; a bit cumbersome but it works. I've also attached the CSV files I used to test it.

Function/S DoLoadMultipleFiles()
    Variable refNum
    String message = "Select one or more files"
    String outputPaths
    String fileFilters = "Data Files (*.txt,*.dat,*.csv):.txt,.dat,.csv;"
    fileFilters += "All Files:.*;"
 
    Open /D /R /MULT=1 /F=fileFilters /M=message refNum
    outputPaths = S_fileName
 
    if (strlen(outputPaths) == 0)
        Print "Cancelled"
    else
        Variable numFilesSelected = ItemsInList(outputPaths, "\r")
        Variable i
        for(i=0; i<numFilesSelected; i+=1)
            String path = StringFromList(i, outputPaths, "\r")
            Printf "%d: %s\r", i, path
           
            //Create names for each column we want
            String TimeName=CleanupName(ParseFilePath(3,path,":",0,0),0)[0,24]+"Time" // the [0,24] needed to allow for 7 characters of "Voltage"
            String VoltageName=CleanupName(ParseFilePath(3,path,":",0,0),0)[0,24]+"Voltage" // the [0,24] needed to allow for 7 characters of "Voltage"
            String WaveNames=""
            sprintf WaveNames,"N='%s';N='%s';",TimeName,VoltageName //compile the wave names into a string the LoadWave/B flag can handle
           
            //Using the /M flag lets us use the /B flag to name the waves properly, but we have to do it one column at a time
            //The critical parameter below is the /L={nameLine, firstLine, numLines, firstColumn, numColumns } flag, so play around with it to get the right behavior
            LoadWave/B=WaveNames/M/D/J/K=1/L={1,2,0,0,1} path //assumes the file has a 1-line header like "Voltage,Time" that we'll ignore
            LoadWave/B=WaveNames/M/D/J/K=1/L={1,2,0,1,1} path //assumes the file has a 1-line header like "Voltage,Time" that we'll ignore
           
            //Since we loaded matrices using /M, let's redimension those into regular 1-D waves
            Redimension/N=-1 $TimeName
            Redimension/N=-1 $VoltageName
        endfor
    endif
 
    return outputPaths      // Will be empty if user canceled
End


I also highly recommend reading up on the built-in BeforeFileOpenHook function so that you can just drag the files of interest into Igor and have them automatically load the way you want (using a one-at-a-time loader instead of the multiple loader we're discussing here). Below is an example that you could modify.
Function BeforeFileOpenHook(refNum,fileName,pathName,type,creator,kind)
    Variable refNum
    String fileName,pathName,type,creator
    Variable kind
    string buffer
   
    Print "------Catching dropped file------"
    Variable handledOpen=0
    if(handledOpen==0)
        if(CmpStr(ParseFilePath(4,fileName,":",0,0),"csv")==0) //check if the file extension is "CSV"
            print "It's a CSV file!"
            print "Assuming we want it loaded by the DoLoadFile routine"
           
            ////It can be useful to use "FSetPos" and "freadline" to check certain text in the file to verify you actually want to load it using the below routine
            //FSetPos refnum, 0
            //freadline refnum, buffer
            ////Then use stringmatch(buffer,"*specialText*") to check if the special text is in the file
           
            DoLoadFile(fileName,pathName) //put your special file loader function name here
            handledOpen=1
        endif
    endif
   
    return handledOpen // 1 tells Igor not to open the file
End

Quote:
Sorry to hijack the thread.


It's better to start a new thread so that one thread addresses one issue.

Quote:
I was wondering if anyone can help with my LoadWave command. The CSVs have two columns: "Time" & "Voltage". I want the output waves to be "CSVfileName"Time and "CSVfileName"Voltage but I can't figure it out.


To control wave names precisely, use the LoadWave /B flag. For example:
String columnInfoStr = ""
columnInfoStr += "N=CVSTime;"
columnInfoStr += "N=CVSVoltage;"
LoadWave/J/B=columnInfoStr ...


For details see "Specifying Characteristics of Individual Columns" in the reference documentation for LoadWave.

In your case, because you want to include the file name in the wave name, you need to compute the values for columnInfoStr.

I would also take the opportunity to split the LoadWave command into a separate function for modularity.

Also, since it appears you are loading numeric data only, not date or string data, I would use /G instead of /J.

You also need to make sure that the wave names are not too long. Igor has a limit of 31 byte at present.

Here is a crack at it, which I have not thoroughly tested:

Function LoadOneFile(pathName, filePath)
    String pathName     // Name of Igor symbolic path or ""
    String filePath     // File name, partial path relative to symbolic path, or full path
   
    String fileName = ParseFilePath(3, filePath, ":", 0, 0) // Get file name without extension
   
    String baseTimeName = "CVSTime_"
    String baseVoltageName = "CVSVolt_"
    Variable baseVoltageMaxNameLength = max(strlen(baseTimeName),strlen(baseVoltageName))
    Variable maxIgorNameLength = 31
   
    // This is to make sure that your wave names don't wind up being too long for Igor to handle
    Variable clippedFileNameLength = maxIgorNameLength - baseVoltageMaxNameLength
    String clippedFileName = fileName[0,clippedFileNameLength-1]
    // Print clippedFileName    // For debugging only
   
    String columnInfoStr = ""
    columnInfoStr += "N='" + baseTimeName + clippedFileName + "';"
    columnInfoStr += "N='" + baseVoltageName + clippedFileName + "';"
    // Print columnInfoStr  // For debugging only
   
    LoadWave/G/D/B=columnInfoStr/K=0/P=$pathName filePath       // You may need to tweak this
End

Function/S DoLoadMultipleFiles()
    Variable refNum
    String message = "Select one or more files"
    String outputPaths
    String fileFilters = "Data Files (*.txt,*.dat,*.csv):.txt,.dat,.csv;"
    fileFilters += "All Files:.*;"
 
    Open /D /R /MULT=1 /F=fileFilters /M=message refNum
    outputPaths = S_fileName
 
    if (strlen(outputPaths) == 0)
        Print "Cancelled"
    else
        Variable numFilesSelected = ItemsInList(outputPaths, "\r")
        Variable i
        for(i=0; i<numFilesSelected; i+=1)
            String path = StringFromList(i, outputPaths, "\r")
            Printf "%d: %s\r", i, path
            LoadOneFile("", path)
        endfor
    endif
 
    return outputPaths      // Will be empty if user canceled
End



Thank you for your help ajleenheer and hrodstein. Both of your solution work and have saved me many evenings of headaches!

Resurrecting the topic! :) 

 

I am using the snippet that ACLIGHT has published above, I cannot get the returned value in my script though...It simply does not appear in my data browser. 

I call it like this:

DoLoadMultipleFiles()


Second question: how should I modify it to return the number of selected files as well?

 

Thank you for help!
 

In reply to by Jakub

In your script you need to call that function and assign the returned string to a string variable.

Function DoStuff()
   String myString = DoLoadMultipleFiles()
   Print myString // or some other command
End

 

Quote:
Second question: how should I modify it to return the number of selected files as well?

You can add a pass-by-reference parameter to DoLoadMultiple and return the number of selected files using it. Execute this for details:

DisplayHelpTopic "Pass-By-Reference"

You can also do it like this:

Function Demo()
    String listOfFullPaths = DoLoadMultipleFiles()
    Variable numItems = ItemsInList(listOfFullPaths, "\r")
    Print numItems
End