Version Control for Igor Analyses

Share icon

Fri, 06/27/2008 - 12:13 pm

The problem of saving many different versions of analysis files and code for a project is one that we probably all experience. After looking at one of my analysis directories and seeing things like "2008_06_10_ConnectivityAnalysis", "2008_05_24_ConnectivityAnalysis", etc., each with their own slightly different versions of the data and different procedure files, I decided to take action.

To some of you this may be a no-brainer, but I started using Subversion (http://subversion.tigris.org/) to address this problem. Subversion is a version control system intended to allow programmers to maintain a repository with a history of all changes to their code, simultaneously accessible and modifiable by multiple users. Different parts of a project can be worked on simultaneously, and reintegrated into the whole, and one can "roll-back" to a previous version at any time.

Subversion's greatest strength is with text files, in which two different versions of a file can be distinguished by saving one copy of the file, and then a list of line-by-line revisions, rather than two whole and mostly similar versions of the file. This saves a lot of disk space, among other things. I realized that when an Igor experiment is saved unpacked, almost all of its components (variables, command history, recreation macros, procedure files, notebooks, but not waves) are essentially just directories of text files. The only "wasted" space in saving multiple versions would be for changes in waves. While this could be substantial, it's certainly no worse than saving a whole new copy of the experiment file.

By placing an unpacked experiment (or a directory of unpacked experiments) under version control, I can have one instance of my analysis, the most updated one, on disk, but roll back to previous versions (stored as a list of changes, on the same disk or a backup server) at any time. This keeps different analysis files that might go together with one another, or with notebooks, etc., all under the same version.

Here are the issues I have identified in using Subversion to provide version control for Igor analyses.
1. Notebooks are recognized as binary files by Subversion, and thus are not given the line-by-line treatment. Change the config file in the Subversion home directory (~/.subversion on Mac) to enable auto-props, and add the auto-prop *.ifn = svn:mime-type=text/ifn. Then notebooks will be treated as text.
2. Subversion doesn't like files with no data fork (Mac only). Delete icon files from Igor experiment folders before importing your project into subversion.
3. All newly created objects will need to be placed under version control. Running svn add * --force, before committing changes, will take care of this. The same applies to newly deleted objects, but you don't really need to delete things in the OS or through Subversion as long as you've deleted it from memory in Igor.

I'm sure there are other issues, but I haven't run into them yet in my testing of this method. If anyone has any insights into this approach to version control for Igor analysis projects, I'd like to hear them. At some point I could even make an Igor procedure with ExecuteScriptText calls, or an XOP, to handle the svn commands.

Rick

RGerkin wrote:
The problem of saving many different versions of analysis files and code for a project ....

Here are the issues I have identified in using Subversion to provide version control for Igor analyses.

1. Notebooks are recognized as binary files by Subversion, and thus are not given the line-by-line treatment. Change the config file in the Subversion home directory (~/.subversion on Mac) to enable auto-props, and add the auto-prop *.ifn = svn:mime-type=text/ifn. Then notebooks will be treated as text.


FWIW, I suspect, the same SVN preference file change might be needed to have HELP files recognized as text.

RGerkin wrote:
...

I'm sure there are other issues, but I haven't run into them yet in my testing of this method. If anyone has any insights into this approach to version control for Igor analysis projects, I'd like to hear them. At some point I could even make an Igor procedure with ExecuteScriptText calls, or an XOP, to handle the svn commands.


Thank you for the insightful presentation.

I would applaud a utility that allows a user to do proper SVN storage of all aspects of an Igor experiment, all the more so if such could be done directly and transparently from within the experiment itself, and even more so if it the approach taken was consistent for both Mac and Windows platforms.

My experiences lead me to suspect, having a consistent approach to the SVN interface will be the hardest target to reach. Already, using SVN on Windows (TortoiseSVN) in far easier than on MacOS (command line + SVNx, + other tools + other tools + ?????). This however does not speak to the underlying system calls needed to execute SVN on each platform, so the inverse may be true when it comes to integrating SVN within Igor Pro. A case in point ... when I use ExecuteScriptText to run XML/XSL on MacOS, all runs consistently, transparently, and directly because XML is built-in to MacOS. On Windows, I have to download an additional application and then watch DOS command-line windows dance across the screen for each ExecuteScript call.

In any event, regardless of how you decide to proceed, any insights you gain would be appreciated.

--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAH

June 27, 2008 at 01:01 pm - Permalink

Rick

This is certainly an interesting idea. I currently keep all of my procedure files under version control using Subversion, but hadn't thought to use unpacked experiments to also keep the rest under version control.

BTW, you may know this already, but project maintainers on IgorExchange also have the option of using Subversion to keep their projects under version control. In conjunction with this feature, I've written up a fairly extensive tutorial describing how to use Subversion on both Windows and MacOS. These tutorials are targeted for IgorExchange project maintainers, but they are applicable to other circumstances as well. See http://www.igorexchange.com/node/67 for more information.

June 27, 2008 at 02:56 pm - Permalink

jjweimer wrote:
The problem of saving many different versions of analysis files and code for a project ....
FWIW, I suspect, the same SVN preference file change might be needed to have HELP files recognized as text.

I believe that Igor help files (.ihf) are actually a binary format, since they can contain pictures and other types of non-text data.

Of course, Subversion can store binary data just as well as text data, but it's difficult to compute a diff between two binary files.

June 27, 2008 at 02:58 pm - Permalink

The same could be said of notebook files, which might contain images. However, my vision is to have my notebook files contain only text, which can be parsed to do image/table/reference insertion, a la LaTeK. I am actually working on the code for this right now, which would allow your entire project, from data to analysis to manuscript, to be under the same Igor file(s), and under the same version control.

aclight wrote:
jjweimer wrote:
The problem of saving many different versions of analysis files and code for a project ....
FWIW, I suspect, the same SVN preference file change might be needed to have HELP files recognized as text.

I believe that Igor help files (.ihf) are actually a binary format, since they can contain pictures and other types of non-text data.

Of course, Subversion can store binary data just as well as text data, but it's difficult to compute a diff between two binary files.

June 30, 2008 at 07:28 am - Permalink