Load and process waves

Share icon

Hello

I try to write a procedure to do:

1 - load all the .CSV file located in a folder (or .txt Space-Delimited Text File)

2 - There is three column In each file, so i have to put the name of the file for the first one (eg fileName), the second : fileName_PM1 and third : fileName_PM2

3- Make the sum (fileName_PM1 + fileName_PM2) and put the result in a new wave fileName_Sum.

For now I managed to achieve the first two points, but for the third one I could not find the correct way to do it !!!! problems with the names of wave / wave reference / string ... ( All the commented part in the end the procedure)

I know this is a little problem but...

is ther a simple solution/function to do the third point ?

Here is the procedure:



Function importAllFiles()


	Variable formatSelector // selector of the file format : 1 for .csv format and  2 for .txt format
	String fileFormat = ""
	Variable  firstLine = 2 // start to load from the line number 2 to skip header 
	Variable  numColumns = 3 //  the number of columns to load.
	
	String extension= "*.csv; *.txt;"
	
	String separator = ""
	
	Prompt formatSelector, "Sélectionnez un format de fichier", popup extension
	
	DoPrompt " File Format ", formatSelector
	if (V_Flag)
		return -1								// User canceled
	endif
	
		
	if ( formatSelector == 1) // 1 for .csv format
	
		fileFormat = ".csv"	
		separator = ";"					
		
	elseif (formatSelector == 2)  // 2 for .txt format
	
		fileFormat = ".txt"	
		separator = "	"	// Space-Delimited Text File			
	
	endif
	
	String messageSt = " Emplacement des fichiers " + fileFormat+ " ? "  //  put the file format in the prompt message.
		  
	NewPath/M=messageSt/O PathName	// get the folder in which the files are
	 
	if (V_flag == -1)   //User cancelled dialog
		return -1
	endif
	 
	string ListOfFiles = IndexedFile(PathName, -1, fileFormat)	// Get list of *.csv files in folder
		
	Variable  nbrOfFiles = itemsinlist (ListOfFiles ) // number of files in the folder 
		
	String fileToLoad 	// name of the file to load
	variable i
		
	String columnInfo = ""
			
	Edit/N=Traitement // Ceat a Table "Traitement" and append the loaded waves to this table
		
	for( i = 0; i
After LoadWave, do this:


String listOfWavesLoaded = S_waveNames		// S_waveNames is created by LoadWave
String firstName = StringFromList(0, listOfWavesLoaded)
Wave firstWave = $firstName
String secondName = StringFromList(1, listOfWavesLoaded)
Wave secondWave = $secondName
String thirdName = StringFromList(2, listOfWavesLoaded)
Wave thirdWave = $thirdName


Now you can use the wave references in subsequent commands.

March 16, 2016 at 10:37 am - Permalink
Thanks a lot hrodstein,

After a few tries...it works.

That may not be the best way to do. The concept of wavename and wave reference is not easy to figure out....

One more question, is there any tips to speed up loading very large files?

many thanks

And here is the new procedure :

 
Function importAllFiles()


	Variable formatSelector // selector of the file format : 1 for .csv format and  2 for .txt format
	String fileFormat = ""
	Variable  firstLine = 2 // start to load from the line number 2 to skip header 
	Variable  numColumns = 3 //  the number of columns to load.
	
	String extension= "*.csv; *.txt;"
	
	String separator = ""
	
	Prompt formatSelector, "Sélectionnez un format de fichier", popup extension
	
	DoPrompt " File Format ", formatSelector
	if (V_Flag)
		return -1								// User canceled
	endif
	
		
	if ( formatSelector == 1) // 1 for .csv format
	
		fileFormat = ".csv"	
		separator = ";"					
		
	elseif (formatSelector == 2)  // 2 for .txt format
	
		fileFormat = ".txt"	
		separator = "	"	// Space-Delimited Text File			
	
	endif
	
	String messageSt = " Emplacement des fichiers " + fileFormat+ " ? "  //  put the file format in the prompt message.
		  
	NewPath/M=messageSt/O PathName	// get the folder in which the files are
	 
	if (V_flag == -1)   //User cancelled dialog
		return -1
	endif
	 
	string ListOfFiles = IndexedFile(PathName, -1, fileFormat)	// Get list of *.csv files in folder
		
	Variable  nbrOfFiles = itemsinlist (ListOfFiles ) // number of files in the folder 
		
	String fileToLoad 	// name of the file to load
	variable i
		
	String columnInfo = ""
			
	Edit/N=Traitement // Ceat a Table "Traitement" and append the loaded waves to this table
		
	for( i = 0; i

March 17, 2016 at 01:39 am - Permalink
You may be able speed up things a bit (depending on how big you data is) by removing the last duplicate, which is unnecessary. Duplicate eats memory and processing power. It is always better to use what is already there.

	for( i = 0; i<nbrOfFiles; i=i+1)	
 			fileToLoad = StringFromList(i, ListOfFiles)	
 			if(strlen(fileToLoad)==0)	// no more file to import
				break
			endif					// Condition;update loop variables
 
			String name = RemoveEnding(fileToLoad, fileFormat ) // remove the .csv from file name
 
			columnInfo += "N='" + name + "';" //  take the name of the file for the firts column
			columnInfo += "N=" + name + "_PM1;"    // add _PM1 for the second column
 			columnInfo += "N=" + name + "_PM2;"    // add _PM2 for the second column
 
			LoadWave/a/E=2/d/j/b=columnInfo/L={0, firstLine, 0, 0, numColumns }/V={separator, "$", 0, 0 }/P=PathName fileToLoad // 	
 			if (V_flag==0)				// No waves loaded. Perhaps user canceled.
				return -1
			endif
 
			String listOfWavesLoaded = S_waveNames		// S_waveNames is created by LoadWave
			String newName = name + "_Sum"

			duplicate/O $StringFromList(0, listOfWavesLoaded) $newName
			Wave waveSum = $newName
 			Wave secondWave = $StringFromList(1, listOfWavesLoaded)
			Wave thirdWave = $StringFromList(2, listOfWavesLoaded)

			waveSum = secondWave + thirdWave
 
			columnInfo = ""	
 	endfor

March 17, 2016 at 03:59 am - Permalink
Thank you chozo for this optimization. My files can be up to some hundred of MB !

March 17, 2016 at 05:44 am - Permalink
is there any tips to speed up loading very large files?


See the section "Loading Very Large Waves" in the help for the LoadWave operation.

Using a solid state hard drive will probably help a lot.

March 17, 2016 at 07:46 am - Permalink
The first duplicate is also not needed, or?
Pass the wavename to LoadWave with 
/N
and tell it to overwrite with 
/O
.

March 17, 2016 at 09:05 am - Permalink
Isn't "loading very large files" already taken care of by the parameter 
/L={0, firstLine, 0, 0, numColumns }
in loadwave or am I missing something?
HJ

March 17, 2016 at 05:15 pm - Permalink
@ HJDrescher : that's what I was wondering and that's why I asked for another tip !!!

@ thomas_braun : I did not really understand what you mean.

March 18, 2016 at 12:43 am - Permalink
Destination wave names can be passed to LoadWave via the /B flag. Syntax is a little bulky: have a look at the /B flag in the help file (and the N parameter therein).
Here's an example for two single precision columns: 
/B=("C=1,F=0,N=CalData; C=1,F=0,N=QMSData;")


HJ
@TB You also got me at first with the /N flag!

March 18, 2016 at 03:08 am - Permalink
Okay, okay. I should have posted a full fledged example ;)


LoadWave/J/N=wave0/O "C:Users:thomas:Desktop:34561_01.csv"


gives

Delimited text load from "34561_01.csv" Data length: 40, waves: wave00, wave01, wave02, wave03, wave04, wave05, wave06, wave07 and: wave08, wave09, wave010, wave011, wave012, wave013, wave014, wave015, wave016 and: wave017, wave018, wave019, wave020, wave021, wave022, wave023, wave024, wave025 and: wave026, wave027, wave028, wave029, wave030, wave031, wave032, wave033, wave034 and: wave035, wave036, wave037, wave038, wave039, wave040, wave041, wave042, wave043 and: wave044, wave045, wave046, wave047, wave048, wave049, wave050, wave051, wave052 and: wave053, wave054, wave055, wave056, wave057, wave058, wave059, wave060, wave061 and: wave062, wave063, wave064, wave065, wave066, wave067, wave068, wave069, wave070 and: wave071, wave072, wave073, wave074, wave075, wave076, wave077, wave078, wave079 and: wave080, wave081, wave082, wave083, wave084, wave085, wave086, wave087, wave088 and: wave089, wave090, wave091, wave092, wave093, wave094, wave095, wave096, wave097 and: wave098, wave099, wave0100, wave0101, wave0102, wave0103, wave0104, wave0105 and: wave0106, wave0107, wave0108, wave0109, wave0110, wave0111, wave0112, wave0113 and: wave0114, wave0115, wave0116, wave0117, wave0118, wave0119, wave0120, wave0121 and: wave0122, wave0123, wave0124, wave0125, wave0126, wave0127, wave0128, wave0129 and: wave0130, wave0131, wave0132, wave0133, wave0134, wave0135, wave0136, wave0137 and: wave0138, wave0139, wave0140

This requires no 
/B
fiddling.

March 18, 2016 at 04:16 am - Permalink
To test my procedure I am trying to load large file.
I have a file that contains a bunch of data about 1.5GB! when I try to load it, charging starts and after a moment : error LoadWave "out of memory" !
what's wrong ?

Here is the procedure :



Function importAllFiles()


	Variable formatSelector // selector of the file format : 1 for .csv format and  2 for .txt format
	String fileFormat = ""
	Variable  firstLine = 2 // start to load from the line number 2 to skip header 
	Variable  numColumns = 3 //  the number of columns to load.
	
	String extension= "*.csv; *.txt;"
	
	String separator = ""
	
	Prompt formatSelector, "Sélectionnez un format de fichier", popup extension
	
	DoPrompt " File Format ", formatSelector
	if (V_Flag)
		return -1								// User canceled 
	endif
	
		
	if ( formatSelector == 1) // 1 for .csv format
	
		fileFormat = ".csv"	
		separator = ";"					
		
	elseif (formatSelector == 2)  // 2 for .txt format
	
		fileFormat = ".txt"	
		separator = "	"	// Space-Delimited Text File			
	
	endif
	
	String messageSt = " Emplacement des fichiers " + fileFormat+ " ? "  //  put the file format in the prompt message.
		  
	NewPath/M=messageSt/O PathName	// get the folder in which the files are
	 
	if (V_flag == -1)   //User cancelled dialog
		return -1
	endif
	 
	string ListOfFiles = IndexedFile(PathName, -1, fileFormat)	// Get list of *.csv files in folder
		
	Variable  nbrOfFiles = itemsinlist (ListOfFiles ) // number of files in the folder 
		
	String fileToLoad 	// name of the file to load
	variable i
		
	String columnInfo = ""
			
	Edit/N=Traitement // Ceat a Table "Traitement" and append the loaded waves to this table
		
	for( i = 0; i

March 21, 2016 at 10:08 am - Permalink
Couple of things.

Avoid the last duplicate command. 

string newName =  name + "_Sum" 
duplicate/O $firstName $newName  // this replaces the first duplicate

should work.
Also have a look at the 
Rename
command.
Is there a reason why you duplicate the fristWave and assign the sumWave as second + third?
Duplication of the second and adding the third might be faster

string newName =  name + "_Sum" // more string declarations
wave sumWave=$newName, thirdWave=$thridName
duplicate/O $secondName $newName  // again, this replaces the first duplicate
sumWave+=thirdwave


Is double precision (~15 digits) necessary or would single (~7 digits) or even integers (maybe coded ["*100"] -- not recommended per se) also work (~half the memory!)? If not, omit the /D flag.

Wave sizes are limited to 2GB in Igor6.x. Depending on your data, this limit might be reached here -- although I doubt it: Your text data would have to be more compact than the binary format in Igor -- like single text-formated bit values stored in double precision later.
Maybe the automatic calculation of the maximal number of rows in /L is causing trouble. Try a reasonable large but constant value. If the error is gone now AND all data is loaded it might be a bug.

@TB /N provides a base name. For this example, we need the /B fiddling. ;-)
"/N=baseName Same as /N except that Igor automatically assigns wave names of the form baseName0, baseName1."

HJ

March 22, 2016 at 09:29 am - Permalink
HJDrescher wrote:
@TB /N provides a base name. For this example, we need the /B fiddling. ;-)
"/N=baseName Same as /N except that Igor automatically assigns wave names of the form baseName0, baseName1."

Yes, you are right. Thanks.

March 22, 2016 at 12:41 pm - Permalink