Load waves from my csv file

I have cleaned up (I think) the data you provided. I think "-6,1.00E-8" should be "1.00E-8".

Also "-6.00E" should be "-6.00E+0" and there was an extraneous "2," in the first line of data.

This gives the following:

X,    CH2,    CH3,    Start,    Increment
Sequence,   VOLT,        VOLT,    -6.00E+0,    1.00E-8
0,    -8.00E,    8.00E-2
1,    8.00E-2,    0.00E+0
2,    -8.00E-2,    0.00E+0
3,    8.00E-2,    0.00E+0
4,    -8.00E-2,    0.00E+0

Given that data, the following code determines the number of data columns to be loaded and loads them:

Function FindNumberOfDataColumns(pathName, fileName)
	String pathName		// Name of an Igor symbolic path or ""
	String fileName		// Name of file or full path to file
	
	Variable refNum

	Open/R/P=$pathName refNum as fileName
	
	String buffer, text
	Variable line = 0
	
	do
		FReadLine refNum, buffer
		if (strlen(buffer) == 0)
			Close refNum
			return -1						// Error: No more data
		endif
		text = buffer[0,7]
		if (CmpStr(text,"Sequence") == 0)
			// This line tells us the number of columns
			int numberOfColumns = ItemsInList(buffer,",")
			int numberOfDataColumns = numberOfColumns - 2	// We want to skip the last two columns
			Close refNum
			return numberOfDataColumns
		endif
		line += 1
	while(1)
	
	return -1		// We will never get here
End

Function FindNumberOfDataColumnsAndLoad(pathName, fileName)
	String pathName		// Name of an Igor symbolic path or "" for dialog
	String fileName		// Name of file or full path to file or "" for dialog

	// First get a valid reference to a file
	if ((strlen(pathName)==0) || (strlen(fileName)==0))
		// Display dialog looking for file.
		Variable refNum
		Open/D/R/P=$pathName refNum as fileName
		fileName = S_fileName			// S_fileName is set by Open/D
		if (strlen(fileName) == 0)		// User cancelled?
			return -2
		endif
	endif

	// Now find the number of data columns in the file
	Variable numberOfDataColumns = FindNumberOfDataColumns(pathName, fileName)
	if (numberOfDataColumns <= 0)
		Print "FindNumberOfDataColumnsAndLoad Error - Could not find number of data columns"
		return -1
	endif
	
	// Now load the data
	int firstDataLine = 2
	LoadWave/Q/J/D/P=$pathName/E=1/K=0/L={0,firstDataLine,0,0,numberOfDataColumns}/N=temp fileName
		
	return 0
End

There are some improvements that I would do:

1. Skip the first column which is of no use assuming that it is always a sequence starting from 0 and stepping by 1.

2. Provide better wave names using the /B flag.

3. Assuming that Start and Increment refer to the starting X value and delta X value, use these to set the X scaling of the loaded waves.

I can help you with those tasks but I'd prefer to work on an actual data file which you can upload as an attachment.

Yes, what I am trying to do is to write a Macro that could load the waves. And the first thing to do is get the starting X value and delta X value, so I could use these to set the X scaling of the loaded waves. But I would like to write a macro that could work for all possible # of channels instead of only two. That is why I am asking how to get the header of last two columns. Thanks

The code above does load all columns of data regardless of how many there are.

Here is an updated version that:

1. Skips the first column which is of no use assuming that it is always a sequence starting from 0 and stepping by 1.

2. Names the output waves CH0, CH1,...

3. Sets the X scaling of each output wave based on the last two numbers in the "Sequence" line.

This code works assuming that the format of the text in the file matches the format of my cleaned-up text in the post above.

If you attach an actual data file, I will test to make sure this code works with your actual data file.

Function ExtractScaling(buffer, x0, dx)
	String buffer		// e.g., "Sequence, VOLT, VOLT, -6.00E, 1.00E-8"
	double& x0			// Output
	double& dx			// Output

	x0 = 123
	dx = 456
	
	String list = ReplaceString(" ", buffer, "")		// Remove spaces
	int numItems = ItemsInList(list, ",")
	String x0Str = StringFromList(numItems-2, list, ",")
	x0 = str2num(x0Str)
	String dxStr = StringFromList(numItems-1, list, ",")
	dx = str2num(dxStr)
	
	String expr = ""
	
	return 0	
End

Function SetWaveScaling(listOfWaveNames, x0, dx)
	String listOfWaveNames
	double x0, dx
	
	int numItems = ItemsInList(listOfWaveNames)
	int itemNumber
	for(itemNumber=0; itemNumber<numItems; itemNumber+=1)
		String name = StringFromList(itemNumber, listOfWaveNames)
		WAVE w = $name
		SetScale/P x, x0, dx, w
	endfor
End

Function FindNumberOfDataColumnsAndScaling(pathName, fileName, x0, dx)
	String pathName		// Name of an Igor symbolic path or ""
	String fileName		// Name of file or full path to file
	double& x0			// Output
	double& dx			// Output
	
	Variable refNum

	Open/R/P=$pathName refNum as fileName
	
	String buffer, text
	Variable line = 0
	
	do
		FReadLine refNum, buffer
		if (strlen(buffer) == 0)
			Close refNum
			return -1						// Error: No more data
		endif
		text = buffer[0,7]
		if (CmpStr(text,"Sequence") == 0)
			// This line tells us the number of columns
			int numberOfColumns = ItemsInList(buffer,",")
			int numberOfDataColumns = numberOfColumns - 2	// We want to skip the last two columns
			Close refNum
			ExtractScaling(buffer, x0, dx)
			return numberOfDataColumns
		endif
		line += 1
	while(1)
	
	return -1		// We will never get here
End

Function FindNumberOfDataColumnsAndLoad(pathName, fileName)
	String pathName		// Name of an Igor symbolic path or "" for dialog
	String fileName		// Name of file or full path to file or "" for dialog

	// First get a valid reference to a file
	if ((strlen(pathName)==0) || (strlen(fileName)==0))
		// Display dialog looking for file.
		Variable refNum
		Open/D/R/P=$pathName refNum as fileName
		fileName = S_fileName			// S_fileName is set by Open/D
		if (strlen(fileName) == 0)		// User cancelled?
			return -2
		endif
	endif

	// Now find the number of data columns in the file
	double x0, dx
	Variable numberOfDataColumns = FindNumberOfDataColumnsAndScaling(pathName, fileName, x0, dx)
	if (numberOfDataColumns <= 0)
		Print "FindNumberOfDataColumnsAndLoad Error - Could not find number of data columns"
		return -1
	endif
	
	// Now load the data
	int firstDataLine = 2
	int firstColumnToLoad = 1		// Skip column 0 which is just 0,1,2,...
	LoadWave/Q/J/D/P=$pathName/K=0/O/A=CH/E=1/L={0,firstDataLine,0,firstColumnToLoad,numberOfDataColumns}/N=temp fileName
	
	SetWaveScaling(S_waveNames, x0, dx)
	
	return 0
End

Much appreciated!

Here is the Macro that I have so far that only works for two channels, could you try to modify that, so it could work for al possible number of channels? I am hesitating to give you my actual data, since it is important.

Macro loadtrace(tracenum,filename,pathname)
//7/22/2024 This macro loads a Rigol scope trace.
	variable/d tracenum // sets naming of waves
	String fileName // name of file to load or "" to get dialog
	String Pathname // name of path or "" get dialog
	variable/d num_waves_loaded
	String listItem
	variable/d start,increment
	String startwavename, incrementwavename
	// Silent 1; PauseUpdate
	String w0, w1, w2, w3, w4 //  Wavenames of loaded waves.
	// can have up to 4 traces, and time is first column
	String newname0,newname1,newname2,newname3,newname4 // New wave names
	// Here is the beginning of an example Rigol file:
	// X,CH2,CH3,Start,Increment
	// Sequence,VOLT,VOLT,-6.00E-6,1.00E-8
	// 0,-8.00E-2,8.00E-2
	// 1,8.00E-2,0.00E+0
	// 2,-8.00E-2,0.00E+0
	// 3,8.00E-2,0.00E+0
	// 4,-8.00E-2,0.00E+0
	// First we have to load the wave header to get the time scale since that is how Rigol formats their csv files:
	//LoadWave/D/J/O/L={0,1,0,3,0}/A=dummy /p=$Pathname fileName
	LoadWave/D/J/O/L={0,1,0,3,0}/A=dummy /p=my_path fileName
	// /L={nameLine, firstLine, numLines, firstColumn, numColumns}
	print("Loaded S_fileName=")
	print(S_fileName)
	print("Loaded S_fileName=")
	print(S_path)
	print(S_waveNames)
	startwavename=StringFromList(0, S_waveNames)
	incrementwavename=StringFromList(1, S_waveNames)
	Pathname=S_path // for next load command, if user selected using dialog
	fileName=S_fileName// for next load command
	// start = ....
	start=$startwavename[0]
	// increment = ...
	increment=$incrementwavename[0]
	print("start=")
	print(start)
	print("increment=")
	print(increment)
	killwaves $startwavename
	killwaves $incrementwavename
	// Next, load in waves.
	print ("loading filanemame=")
	print filename
	//LoadWave/A=dummy/Q/D/G/O/p=my_path filename
	LoadWave/D/J/O/L={0,2,0,0,3}/W/A/Q /p=my_path fileName
	if (V_flag==0) // Now waves loaded. Perhaps user cancelled.
		return
	endif		
	print("Loaded S_fileName=")
	print(S_fileName)
	print("Loaded S_fileName=")
	print(S_path)
	print("S_waveNames=")
	print(S_waveNames)	
	//Print ItemsInList(stringList, ",")
	print("num_waves_loaded=V_flag=")
	print(V_flag)
	print("done printing....")
	w0= StringFromList(0,S_waveNames) // this is the point number, we don't want it
	w1= StringFromList(1,S_waveNames)
	w2= StringFromList(2,S_waveNames)
	w3= StringFromList(3,S_waveNames)
	w4= StringFromList(4,S_waveNames)
	// put the name of waves into string variables
	if (strlen(w0) != 0) // w0 exists
		print("w0 ok!")
		w0= StringFromList(0,S_waveNames) 
		killwaves $w0 // this is the point number, we don't want it
	endif
	if (strlen(w1) != 0) // w1 exists
		print("w1 ok!")
		w1= StringFromList(1,S_waveNames)
		newname1=w1+"_" + num2str(tracenum)
		//newname1="data_1_" + num2str(tracenum)
		duplicate/o $w1, $newname1
		killwaves $w1
		// set scale to volts		
		SetScale/P x start,increment,"s", $newname1
		SetScale d 0,0,"V", $newname1
	endif
	if (strlen(w2) != 0) // w2 exists
		print("w2 ok!")
		w2= StringFromList(2,S_waveNames)
		newname2=w2+"_" + num2str(tracenum)
		//newname2="data_2_" + num2str(tracenum)
		duplicate/o $w2, $newname2
		killwaves $w2
		// set scale to volts		
		SetScale/P x start,increment,"s", $newname2
		SetScale d 0,0,"V", $newname2
	endif
	if (strlen(w3) != 0) // w3 exists
		print("w3 ok!")
		w3= StringFromList(3,S_waveNames)
		newname3=w3+"_" + num2str(tracenum)
		//newname3="data_3_" + num2str(tracenum)
		duplicate/o $w3, $newname3
		killwaves $w3
		// set scale to volts
		SetScale/P x start,increment,"s", $newname3
		SetScale d 0,0,"V", $newname3
	endif
	if (strlen(w4) != 0) // w4 exists
		print("w4 ok!")
		w4= StringFromList(4,S_waveNames)
		//newname4="data_4_" + num2str(tracenum)
		newname4=w4+"_" + num2str(tracenum)
		duplicate/o $w4, $newname4
		killwaves $w4
		SetScale/P x start,increment,"s", $newname4
		SetScale d 0,0,"V", $newname4
		// set scale to volts		
	endif
EndMacro

I wonder, did you look at what hrodstein has provided? Did it not work for you? It seems a very neat solution to me... Also, macros are very old tech and should be avoided in favor of functions.

Anyway, here is a code fragment which wraps your manual assignment into a simple loop. Things could be done much more neatly (see Howards solution), but if you just want to quickly modify what you already have...

... code until the main load command ...

	LoadWave/D/J/O/L={0,2,0,0,3}/W/A/Q /p=my_path fileName
	if (V_flag==0)
		return
	endif
	Variable i
	for (i=1; i<ItemsInList(S_waveNames);i+=1)
		string currW = StringFromList(i,S_waveNames)
		string Wname = currW + "_" + num2str(tracenum)
		Duplicate/o $currW, $Wname
		KillWaves $currW
		SetScale/P x start,increment,"s", $Wname
		SetScale d 0,0,"V", $Wname
	endfor

... whatever else you want to do ...

minghaoj wrote:
 

   LoadWave/D/J/O/L={0,2,0,0,3}/W/A/Q /p=my_path fileName

Agree to chozo's take. It really comes down to the /L={0,2,0,0,3}. The 3 dictates how many columns you are loading (in your case the X, CH2, and CH3). You can increase this number accordingly if you have additional channels of data. Since your meta data (Start and Increment) are in the first rows, you won't have issue loading your data.

I am guessing your question is how to have your function be versatile to handle things beyond (the repetitive chunks of code for) w0 to w4??

Personally, for data like this, I prefer to load them into a 2D wave matrix by adding /M to LoadWave and go from there.

Another option would be creating a function that does your post loading processing and then feed the waves generated to that function in a loop. I see you already found the list of names of waves are stored in S_waveNames. You can use ItemsInList to determine the number of channels/waves loaded. Then use a For loop to repeat the  process.

int i, numWaves
string wvName, newWvName
numWaves = itemsinlist(S_waveNames)
for(i =0; i< numWaves;i++)
    wvName= StringFromList(i,S_waveNames)
    newWvName = wvName + "_" + num2str(tracenum)
    duplicate/o $wvName, $newWvName
    killwaves $wvName
    // set scale to volts
    SetScale/P x start,increment,"s", $newWvName
    SetScale d 0,0,"V", $newWvName
endfor

Personally, I would change the order of operation as follow and use name the waves differently (you can also use the LoadWave/Name added in Igor Pro 9 or later to handle your wave name changing. Note to myself, start using this).

int i, numWaves
string loadedWvName, wvName
numWaves = itemsinlist(S_waveNames)
for(i =0; i< numWaves;i++)
    loadedWvName= StringFromList(i,S_waveNames)
    // set scale to volts
    SetScale/P x start,increment,"s", $loadedWvName
    SetScale d 0,0,"V", $loadedWvName
    //Chose one below
    //If you prefer the same channel of data from different files are grouped together in Data Browser use this line
    wvName = "CH" + num2str(i+1) + "_" + num2str(tracenum)
    //If you prefer data of different channels from the same data file are grouped together in Data Browser use this line.
    wvName = "_" + num2str(tracenum) + "_CH" + num2str(i+1) //Add "_" in the front as you can't start a wave name using a number, unless you are doing liberal name. The next line of code will fix this for you anyways though.
    //It is always important to verify your wave name is valid. 
    //Alternatively, you can use CreateDataObjectName for more options only if you are using Igor Pro 9.0 or later.
    wvName = CleanupName(wvName, 0) 
    duplicate/o $loadedWvName, $wvName
    killwaves $loadedWvName
endfor

Unrelated, yet related. I have created loaders for several different types of data files with meta data in it. The solution is similar to what hrodstein provided here. It involves a combination of FReadline and LoadWave (sometimes multiple of each).

Please correct me if I am wrong about this. One pet peeve of mine is that LoadWave/L seems to be limited by the number of columns in the firstLine of data which overrides the numColumns parameter if the later is larger. I am sure there are underlying reason why it does this, but part of me hope we can have an option to override that.

The situation is the worst when multiple chunks of data are stored in the same file with their individual meta data. Doing multiple loops of LoadWave/L can be very slow. FReadline appears to be faster (as the file is only opened once) but requires additional post loading wave handling. Some times I can try loading the entire file into one wave and separate each chunk out. It gets messy if the chunk sizes are dynamic.

One pet peeve of mine is that LoadWave/L seems to be limited by the number of columns in the firstLine of data which overrides the numColumns parameter if the later is larger.

That's correct. It is a limitation that I have run into myself on occasion.

I am sure there are underlying reason why it does this

The underlying reason is that I didn't think of this situation when I wrote this code (about 35 years ago ;)

but part of me hope we can have an option to override that.

I think a flag that says "don't automically determine the number of columns in the file from the first line of data, follow /L instead" is doable.

hrodstein wrote:

I think a flag that says "don't automically determine the number of columns in the file from the first line of data, follow /L instead" is doable.

That will be cool. I am looking forward to that feature in future updates.