Translate Spreadsheet Commands to MatrixOP?
jjweimer
Sat, 04/09/2016 - 10:36 am
Here is something that can be done in a spreadsheet with no effort: average and standard uncertainties in rows of data. Here is what I cobbled together using what I know as the standard.
Function DoCalcs()
wave rtimes0, rtimes_ave, rtimes_std
variable ic
for (ic=0;ic<5;ic+=1)
make/N=3/FREE ww
ww = rtimes0[ic][p]
WaveStats/Q ww
rtimes_ave[ic] = v_avg
rtimes_std[ic] = v_sdev
endfor
return 0
end
wave rtimes0, rtimes_ave, rtimes_std
variable ic
for (ic=0;ic<5;ic+=1)
make/N=3/FREE ww
ww = rtimes0[ic][p]
WaveStats/Q ww
rtimes_ave[ic] = v_avg
rtimes_std[ic] = v_sdev
endfor
return 0
end
How would this be done in Igor Pro using MatrixOP?
April 11, 2016 at 01:37 pm - Permalink
Welcome to MatrixOP :)
The main thing to remember about this operation is that it is "matrix-centric" (i.e., focuses on 2D data) but it also handles all 1D waves as a special case of Nx1 matrices. It handles 3D and 4D on a layer-by-layer basis, and it has some special operations that extract smaller dimensional subsets from 3D and 4D waves.
Step 1: skip the make inside the loop and re-write:
ww = rtimes0[ic][p]
as:
MatrixOP/O/FREE ww=row(rtimes0,ic)so MatrixOP creates a wave that contains the row data and you do not need the separate Make command.
Step 2:
rtimes_ave[ic] = v_avg
rtimes_std[ic] = v_sdev
So it seems that you want to average over rows and compute the stdv.
You have several options here depending on what you might want to do with intermediate values. If you wanted to compute these one at a time (not recommended), you could use the MatrixOP mean() function.
The most efficient MatrixOP translation is to skip the loop completely using the following three lines:
MatrixOP/O avg=averageCols(aa)
MatrixOP/O var=sqrt(varcols(aa))
I hope this helps,
A.G.
WaveMetrics, Inc.
April 11, 2016 at 03:44 pm - Permalink
This is the same question I asked last time about MatrixOP (http://www.igorexchange.com/node/5674), and the answer (http://www.igorexchange.com/node/5674#comment-9192)
April 11, 2016 at 08:17 pm - Permalink
Yes, AG, MatrixOP is now on my radar to learn. :-)
I might be so bold as to ask that averageRows, averageCols, varRows, and varCols become standard expressions in MatrixOP's lexicon. While I know how a transpose works, folks who use spreadsheet calculations for just the basics typically do not. For them (and me too), it would be nice to forego the "thinking about it" part and just be able to write ...
MatrixOP/O var=sqrt(varRows(rtimes0))
I might also be even bolder and wonder whether someday we might write this for MatrixOP (and other operations that might be done in sequences) ...
avgR = averageRows(...)
varR = sqrt(varRows(...))
avgC = averageCols(...)
varC = sqrt(varCols(...))
UNINVOKE MatrixOP
This would make a compilation of "spreadsheet-like" functions easier to document and teach.
--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAH
April 16, 2016 at 08:51 am - Permalink
I will plead guilty; I have not used spreadsheet programs in about 20 years so their terminology is not exactly familiar. Also, when I designed MatrixOP, spreadsheet functionality and terminology where not even a remote consideration; I was just trying to provide a powerful tool for advanced users.
I am not convinced that the syntax above has any benefits besides the fact that you are factoring out MatrixOP. I think it helps to remember that MatrixOP is not just another way to compute wave1+wave2. It has significance as a Matrix operation and for that it seems useful to designate the command as such. If that is not immediately obvious, consider what it means to execute
when wave1 is 3D and wave2 is 2D.
A.G.
WaveMetrics, Inc.
April 18, 2016 at 05:15 am - Permalink
AG ... Perhaps we should take this discussion to a side one-on-one thread. In a nutshell, I think that bringing in some basic spreadsheet terminology to Igor Pro would really enhance it in two ways. First, it would offer a stronger side-by-side competition between Igor Pro and "that other analysis program" whose Origin is based on a spreadsheet paradigm. Secondly, it really will offer folks like me who teach undergraduate courses a stronger platform to advocate for Igor Pro as the best-next-step in programming tool after a spreadsheet package (rather than MatLab or MathCad or Mathematica or python or GNUplot or ... not really much of anything else).
I am drawing on analogies to Maple. When a particular set of sequential operations requires a particular package, the syntax used is to start with a statement > with(_package_ ) as a predicate to the full set of steps that follow. I offer this only as a case to collapse the syntax of MatrixOP for a set of code lines, not to defeat the functionality of MatrixOP. And, I offer it perhaps as a thought for Igor Pro 8, not Igor Pro 7. :-)
--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAH
April 18, 2016 at 09:05 am - Permalink
This operation invokes spreadsheet-equivalent commands on a 2-D matrix of data.
Commands return a 1-D wave of values in the result. Many of the commands supported in the
Spreadsheet operation have direct analogs in other functions or operations in Igor Pro.
The commands are reproduced here as well to give users who know spreadsheet commands
a clearer translation of their common syntax when using Igor Pro.
The syntax of the Spreadsheet operation is
result — a 1-D wave
command — an function command call (see list below)
matrix — a 2D wave
direction — the specific term rows or cols
Commands that are currently supported are below.
They have a literal one-to-one translation to common spreadsheet applications.
average
stdev
sum
count
Example
Spreadsheet/Z result = average(dailyruns,cols) // returns the average across columns of dailyruns
--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAH
April 18, 2016 at 09:38 am - Permalink
Glad to do so when I return to the office. I'm on vacation in Korea at the moment.
AG
April 19, 2016 at 04:50 am - Permalink
I made some simple functions that make use of the PCST flag to achieve the desired result.
On a 64 bit windows machine, I placed the attached file in the following folder:
C:\Program Files\WaveMetrics\Igor Pro 7 Folder\Igor Procedures
Once that file is in place, open up Igor and you have direct access to call the functions:
average
count
stdev
sumalong (sum was already taken)
Try it:
average(testdata)
If I understood what you are looking for, this already gives the desired result without waiting for Igor 8, or having to invoke some extra package, or even mess with XOPs.
This will only operate on column data, and of course the point is well taken that for higher dimensions things get complicated, but I think it handles what you are looking for.
I am not sure if it is well advised or not, but I definitely do this sort of thing with some of my favorite custom functions - another selling point of Igor Pro in my mind.
April 19, 2016 at 12:13 pm - Permalink
That's a good point. There's also the /RMD flag that allows you to specify a subset of the data that should be considered in the more familiar [rows][cols]... subrange specification.
For the record, we recommend that you place things like this in the Igor Procedures directory within your Igor Pro User Files directory, not in Igor's own folder. For more information, execute:
April 20, 2016 at 05:37 am - Permalink
* The return for a spreadsheet operation is "hidden". For example, where do you get the results after you do the
average(...)function?* Translations of the syntax from a spreadsheet to Igor Pro is not exactly preserved in all cases. The change
sumalong(...) <- sum(...)requires an extra step and may not be intuitive to everyone (why notsumrange(...)for example?).* Folks have to load extra stuff
I suggest that, when considering the inclusion of spreadsheet options in Igor Pro, the discussion must start from this question: Does this method support the ability for a NOVICE USER to step in to Igor Pro and "see" spreadsheet commands. By novice, I mean "anyone who has essentially no clue how to program anything at all". The translation for spreadsheet -> Igor Pro has got to be as easy as possible for them; it has got to be "oh, I see how to do this", not "oh, I'll have to _learn_ how to do this". Otherwise, novice users will drop it and go on to something else. I also suggest that, the development should be folded in to Igor Pro as though it is a built-in "package". IOW, it should be wrapped in an in-built operation
Spreadsheet .... This is a convention that "sticks" for novice users and seasoned programmers alike.I see that I missed the option to set subranges, as can be done in a spreadsheet. Food for further thought.
--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAH
April 20, 2016 at 06:22 am - Permalink
I thought I'd chime in at this point to say: I know exactly what you mean. However, I think that spreadsheets and Igor are diammetrically opposed. When showing somebody Igor for the first time. I explain that a Table is not like a spreadsheet and explain about 1D waves. The ipf posted above is a good workaround for this use case IMO.
April 20, 2016 at 08:34 am - Permalink
I'd like to be able to teach novices (my undergraduate students) that "Igor Pro is an extension starting from and going beyond spreadsheets" rather than "Igor Pro is diametrically opposed to a spreadsheet program". I agree, a data table in Igor Pro is NOT a spreadsheet, and I would not argue that it ever should be (that can be "the Origin way"). Finally, I know for certain that novices (again ... my undergraduate students) will see the posted ipf as a workaround in the same way that construction workers might see the back end of a screwdriver can be a hammer in a pinch, but then they will just go get a hammer.
Here's the instructions I would love to give to a novice ...
* Use the load data menu to import your data from Excel in to Igor Pro. Put the data in a wave called mydata
* On the command line, type these four commands ...
-- Spreadsheet/AUTO colaverage = average(mydata[][*])
-- Spreadsheet/AUTO colstdev = stdev(mydata[][*])
-- Display colaverage
-- Modifygraph ... // add colstev as the error bars
* You now have the starting point to make a publication-ready graph. See, wasn't that so much faster than mucking about in Excel? :-)
--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAH
April 20, 2016 at 10:55 am - Permalink
If you're using Igor 7, it's actually that easy:
WaveStats/PCST mydata
Duplicate M_WaveStats, mydata_stats
Display mydata_stats[%avg][]
ErrorBars mydata_stats Y,wave=(mydata_stats[%sdev][*],mydata_stats[%sdev][*])
The Duplicate line isn't actually required (if you don't use it, just replace mydata_stats with M_WaveStats in all commands), but it's recommended so that the data doesn't get overwritten if the user later wants to repeat the analysis on a different set of data.
April 20, 2016 at 02:04 pm - Permalink