Loading multiple files and calculating average hourly

Hi all,
I am new to IGOR. I just started to learn Igor instead of Origin.

I have data (tons!!) of 1 sec concentration of chemical species (CO2 and CH4) over a couple of year.. And I would like to calculate mean and standart deviation for each hour over the length of my data.

I have tons of files (*dat), but I can't load all using Igor. I tried the function LoadAndGraphAll as suggested in manual, but I failed. I think that the problem is because I have a lot of rows and columns!! There are a lot of columns, but I need load just DATE TIME CH4_dry CO2_dry

Any help would be greatly appreciated as I am struggling with this.

CO2example.txt (462.16 KB)
I am new to IGOR.

If you have not already done it, do the guided tour. It is essential. Choose Help->Getting Started.

Your file is a FORTRAN-style fixed field file with 23 columns in fields of 26 characters each. Therefore we need to use LoadWave/F.

After you do the guided tour you will need to learn about Igor data folders. Execute:

DisplayHelpTopic "Data Folders"


I have pasted below an Igor procedure that will load your file and another that will load all .txt files in a given folder on disk into Igor data folders. Execute this to learn about Igor procedure files:


DisplayHelpTopic "Procedure Windows"


You should paste the procedure text below into an Igor procedure file and save the file in your "Igor Procedures" folder. The "Igor Procedures" folder is explained in the help for procedure windows.

The procedures add items to the Load Waves submenu of the Data menu.


#pragma rtGlobals=3		// Use modern global access method and strict wave access.

Menu "Load Waves"
	"Load Nogueira File...", LoadNogueiraFile("", "")
	"Load All Nogueira Files in Folder...", LoadAllNogueiraFilesInFolder("")
End

Function LoadNogueiraFile(pathName, fileName)
	String pathName		// Name of an Igor symbolic path or "".
	String fileName			// Name of file or full path to file.
 
	// First get a valid reference to a file.
	if ((strlen(pathName)==0) || (strlen(fileName)==0))
		// Display dialog looking for file.
		Variable refNum
		Open/D/R/F="*.txt"/P=$pathName refNum as fileName
		fileName = S_fileName			// S_fileName is set by Open/D
		if (strlen(fileName) == 0)		// User cancelled?
			return -1
		endif
	endif
 
	// Now load the data.
	// The file uses FORTRAN fixed-field style with 23 columns in fields of 26 characters each.
	// We are interested in the DATE, TIME, CH4_dry, and CO2_dry columns only.

	// "Date" and "Time" are not available as wave names because they are Igor function names
	String columnInfoStr = " "
	columnInfoStr += "N='DateTimeW',F=6;"		// Load DATE column - will become date/time wave
	columnInfoStr += "N='TimeW',F=7;"			// Load TIME column
	columnInfoStr += "N='_skip_';"				// Skip FRAC_DAYS_SINCE_JAN1
	columnInfoStr += "N='_skip_';"				// Skip FRAC_HRS_SINCE_JAN1
	columnInfoStr += "N='_skip_';"				// Skip EPOCH_TIME
	columnInfoStr += "N='_skip_';"				// Skip ALARM_STATUS
	columnInfoStr += "N='_skip_';"				// Skip AmbientPressure
	columnInfoStr += "N='_skip_';"				// Skip CH4
	columnInfoStr += "N='CH4_dry';"			// Load CH4_dry
	columnInfoStr += "N='_skip_';"				// Skip CO2
	columnInfoStr += "N='CO2_dry';"			// Load CO2_dry
	columnInfoStr += "N='_skip_';"				// Skip CavityPressure
	columnInfoStr += "N='_skip_';"				// Skip CavityTemp
	columnInfoStr += "N='_skip_';"				// Skip DasTemp
	columnInfoStr += "N='_skip_';"				// Skip EtalonTemp
	columnInfoStr += "N='_skip_';"				// Skip H2O
	columnInfoStr += "N='_skip_';"				// Skip MPVPosition
	columnInfoStr += "N='_skip_';"				// Skip OutletValve
	columnInfoStr += "N='_skip_';"				// Skip WarmBoxTemp
	columnInfoStr += "N='_skip_';"				// Skip cavity_pressure
	columnInfoStr += "N='_skip_';"				// Skip cavity_temperature
	columnInfoStr += "N='_skip_';"				// Skip solenoid_valves
	columnInfoStr += "N='_skip_';"				// Skip species
	columnInfoStr += "N='_skip_';"				// Skip time
	LoadWave /F={23,26,0} /L={0,1,0,0,0} /D /K=0 /A /B=columnInfoStr /Q /R={English, 1, 2, 1, 1, "Year-Month-DayofMonth", 40} /P=$pathName fileName
	Variable numWavesLoaded = V_flag			// V_flag is set by LoadWave
	if (numWavesLoaded != 4)
		Print "Error loading file - wrong number of waves loaded"
		return -1
	endif
 
	Wave DateTimeW,TimeW			// Create reference to waves created by LoadWave
	DateTimeW += TimeW				// Add time to date to obtain date/time
	KillWaves/Z TimeW					// This is no longer needed

	return 0							// Success
End

Function LoadAllNogueiraFilesInFolder(pathName)	// Loads all .txt files in specified folder
	String pathName					// Name of symbolic path or "" to get dialog
	String fileName
	Variable index=0

	if (strlen(pathName)==0)			// If no path specified, create one
		NewPath/O temporaryPath		// This will put up a dialog
		if (V_flag != 0)
			return -1					// User cancelled
		endif
		pathName = "temporaryPath"
	endif

	Variable result
	do			// Loop through each file in folder
		fileName = IndexedFile($pathName, index, ".txt")
		if (strlen(fileName) == 0)			// No more files?
			break									// Break out of loop
		endif

		// Create data folder from file name
		String dfName = ParseFilePath(0, fileName, ":", 1, 0)
		dfName = RemoveEnding(dfName, ".txt")	
		if (DataFolderExists(dfName))
			String prompt
			sprintf prompt, "Data folder named '%s' exists. Click Yes to overwrite the existing data, No to cancel.", dfName
			DoAlert 1, prompt
			if (V_flag == 2)
				Print "Load cancelled. You can rename the existing data folder and try again."
				return -1			// Cancelled
			endif
			SetDataFolder $dfName
		else
			NewDataFolder/S $dfName
		endif
		
		result = LoadNogueiraFile(pathName, fileName)

		SetDataFolder ::				// Back to parent data folder

		index += 1
	while (1)

	if (Exists("temporaryPath"))		// Kill temp path if it exists
		KillPath temporaryPath
	endif

	return 0						// Signifies success.
End
Many thanks hrodstein!!

This is a great start. I've made the tour#1 and #2... but everything is new for me.

Best,