Loading waves from "ragged" CSV files (Help!)

You did not provide an example file, but pasting the numbers into a text file and using your LoadWave command on it loads the data just fine exactly as you want. You do not post what you input as columnInfoStr, so it is impossible to give advice here. Maybe you are confused by the treatment of rows and columns which is a bit different to some other programs. Igor is a column-major application. What is the issue here, i.e., what does not work for your?

Use FReadLine:

Function LoadRaggedCSV(pathName, fileName, firstLine, numCols)
	String pathName		// Igor symbolic path name, or "" for dialog
	String fileName		// File name or full path, or "" for dialog
	Variable firstLine	// 1 is the first line in the file
	Variable numCols

	Variable refNum
	Open/R/P=$pathName/Z=1 refNum as fileName
	if (V_flag != 0)
		return -1
	endif

	//String colNames = "A;B;C;D;E;F;G;H;I;J;K;L;M;N;O;" // e, p, q, r, s, and x, y, z, t should be avoided
	String colNames = "A0;B0;C0;D0;E0;F0;G0;H0;I0;J0;K0;L0;M0;"
	if( numCols > ItemsInList(colNames) )
		Abort "Too many columns for colNames"
		return -1
	endif

	// First pass: count lines with data
	Variable numRows = 0
	String line
	Variable lineNum = 1
	do
		FReadLine refNum, line
		if (strlen(line) == 0) // note: a blank line will have at least a line ending \n or \r
			break // EOF
		endif
		// a blank line will put NaN in each column, so we assign a wave row for it
		if (lineNum >= firstLine)
			numRows += 1
		endif
		lineNum += 1
	while (1)

	// Make the waves
	Variable i
	for (i = 0; i < numCols; i += 1)
		Make/O/D/N=(numRows) $StringFromList(i, colNames) = NaN
	endfor

	// Second pass: fill
	FSetPos refNum, 0
	Variable row = 0
	lineNum = 1
	do
		FReadLine refNum, line
		if (strlen(line) == 0)
			break // EOF
		endif
		if (lineNum >= firstLine)
			line = ReplaceString("\r", ReplaceString("\n", line, ""), "")
			for (i = 0; i < numCols; i += 1)
				String item = StringFromList(i, line, ",")
				if (strlen(item) > 0)
					Wave w = $StringFromList(i, colNames)
					w[row] = str2num(item)
				endif
			endfor
			row += 1
		endif
		lineNum += 1
	while (1)

	Close refNum
	return 0
End

•LoadRaggedCSV("home", "RaggedDelimitedTextData.txt",2, 6) // load starting at second line, assume 6 columns

•Edit/K=0 root:A0,root:B0,root:C0,root:D0,root:E0,root:F0

The file I created from your example

The problem with LoadWave/J (Load Delimited Text) is that it determines the number of columns to load from the first line to be loaded. It also expects that there is a delimiter between each column. So, if your first line loaded is:

1,2,3

LoadWave will create three waves. To make it create 5 waves, you could change the first line to:

1,2,3,,

You don't need to change all of the lines, just the first one to be loaded.

Your LoadWave command uses /L={0,1,0,0,0}. The "1" means that the first line to be loaded is line 1 which is the second line in the file (since Igor uses 0-based numbering). Unless there is a blank line in the file above the first line of data, you can omit the /L flag.

Jim ... Off the top of my head, I wonder if you need two FReadLine passes. At the first FReadLine pass, read each line into a (one column) text wave. End with a text wave having as many rows Nr as the number of rows appearing in the CSV. Do the rest of the processing internally. Create a wave nValueswave with Nr to hold how many text entries are in each slot of the text wave. Find the maximum number in nValueswave -> This is the number of columns Nc to create. Create a matrix Nr x Nc with NaN entries. Fill only those slots needed in the matrix from the text wave using the entries in nValueswave. Finally, split the matrix to Nc waves of length Nr.

jjweimer wrote:

Jim ... Off the top of my head, I wonder if you need two FReadLine passes. At the first FReadLine pass, read each line into a (one column) text wave. End with a text wave having as many rows Nr as the number of rows appearing in the CSV. Do the rest of the processing internally. Create a wave nValueswave with Nr to hold how many text entries are in each slot of the text wave. Find the maximum number in nValueswave -> This is the number of columns Nc to create. Create a matrix Nr x Nc with NaN entries. Fill only those slots needed in the matrix from the text wave using the entries in nValueswave. Finally, split the matrix to Nc waves of length Nr.

You are right, of course. Mine was just a simple working example of how it could be done.

when loadwave doesn't offer a direct solution, i might go about it like this:

use grep to dump file into a textwave:

Make/T/N=0/free textwave
Grep/E="" filePath as textwave

make a 2D wave with/N=(numpnts(textwave), numcolumns)

set wave2d = stringfromlist(q, textwave[p], ",")

use splitwave to create output waves

it's not tidy and it's not efficient, just another way to do the same thing.