Spectra WatchDog - On-the-fly and interactive data logger
Thu, 07/27/2023 - 11:36 pm
If you want to create data logs while (or after) measuring data with the ability to edit entries on the fly, then this package is for you.
Spectra WatchDog watches a specified folder for new data files of a specified extension, and then records specified metadata in a log entry. This is most useful if your measurement software dumps finished files in a single folder, which will then be automatically processed. Alternatively, a log can be created from a folder of prerecorded data to generate an annotated summary. The log can be edited (e.g., adding notes or changing entries) in an user interface within Igor Pro. The parsing function is provided by the user (see below). Currently, a function for loading Scienta-Omicron SES files (*.ibw format) is provided with the package. Support for zipped data is provided as well (the data will be temporarily unzipped in this case). Log files are simple tab-delimited text files with the ending *.log (the ending can be changed in desired).
I tried my best to test everything thoroughly, but sometimes bugs slip through. Bug reports and suggestions for new features are always welcome. I am happy to add definitions for popular data formats and open to feature requests.
The User Interface
After installing the package two new options appear under the menu entry Spectra Tools: Folder Watchdog and Folder Quick-log; the latter will be explained later. After selecting Folder Watchdog the user interface appears:
First, there are a few things to set up:
- Folder to Watch: Specify the folder to be watched for new data files or to read data from.
- Save Log File to: Specify where the log is written to. This file can be saved anywhere you have write permission.
- Press Read Log Entries if you want to read the entries from the specified (existing) log into the table below. This is useful if you continue working with an existing log file.
- File Type(s): Specify the file ending(s) to watch for. You need to provide at least one loading function for the specified file ending (see below). Multiple file types can be specified as a semicolon separated list.
- Set Ignore Existing Files: This will ignore all currently available files in the folder and only checks for newly added files.
Now you can press either:
- Start Watchdog: The program logs all currently available files with the correct ending and then keeps watching for new files. If you want to stop the process, press Stop Watchdog.
- Read Folder Once: The program just logs all currently available files.
The remaining buttons are:
- Reset All: Clears the table and resets the list of already processed files in the folder. Use this to start anew. You might want to specify a new log as well. If you don't specify a new log file, then files already logged in the file will not be reprocessed, since you cannot have the same data entry in the log twice.
- Clear Display List: Clears all entries in the table below (NOT in the log file). Use this if the table got very long and you want to concentrate only on new entries.
NOTE: Any setting you make will be saved in a file named 'Spectra WatchDog settings.dat' next to the procedure file. This is a simple text file, which can also be edited with a text editor.
Here is how above log file looks in Excel:
CAUTION: Do not open the log file in Excel (or any other program that locks write access to the file) at the same time while you are still working with it in the Watchdog. Igor cannot update the log in that case.
Working with the Table
Logged entries can be directly edited in the table. Simply click a cell to edit it's contents. Editing any cell with also update the log file on the fly. In other words, the contents of the log and the table are synced. The cell in the first column is highlighted for each selected row. The first column is meant to serve as a data tag for identifying the data entry.
Cell selection:
Select single cell with the mouse. Holding shift or cmd / ctrl will select multiple consecutive or disjointed cells respectively. ctrl/cmd+a will select everything (be careful not to delete everything in this case).
Column width and color:
Drag the divider lines in the column header to adjust the cell width of this column. Right-clicking on the cell header allows you to select a color for the column. Both settings will be remembered.
Column options:
In the same, right-click menu of the cell header are additional options:
Widths:
- Default: The default width of the package (optimized for the included file logger).
- Optimal: Tries to adjust the columns to allow for optimal spacing depending on the current contents in each column.
- Equal: All widths will be the same.
Repeat Values option:
If a new entry is written, the content of the previous entry will be prefilled for this column. Use this setting for columns which contain mostly the same information which you do not want to insert repeatedly.
Cell options:
Right-clicking on a cell when not in edit mode, reveals some common options:
Keyboard shortcuts:
- cmd/ctrl + a = select all
- cmd/ctrl + x = cut
- cmd/ctrl + c = copy
- cmd/ctrl + v = paste
- cmd/ctrl + z = undo/redo
- del key = delete
- cursor keys = move selection
Writing a Parsing Function
The most important part for using this package with your own data is to write a function with parses the meta data from a file. You can parse as few contents as you like, but the more information is automatically gathered from the files the easier is the logging process on the user.
The parsing function need to have the following format:
...
return textWave // two columns required: [][0] = entry name, [][1] = value
end
Thus, the function name needs to start with 'WDfunc_' to be recognized by the tool, followed by some EXTension: This must be the file extension, such as 'txt', 'ibw', 'tiff', 'dat' etc. you are going to support with this function. You can have multiple functions for the same extension, if you have different types of files which share the same extension or you can process all these different types in a single function. Finally, the function ends with a freely choosable name.
The filePath string is the full path to one file on disk which should be parsed. The task of this function is to give back a two-column text wave with the desired metadata parsed from the file (and some extra rows for manual information if desired). This can be a free wave. The first column of this wave is the title of the information, which will provide the table's column titles. The second column is the information itself. If your parser encounters the wrong file return an invalid wave ($"") to indicate that this file is not suitable and should be ignored. The logger will then attempt to use another function for this extension (if available) until the file could either successfully be parsed or the file is ignored for good.
As an example, here is a parser which will read text files (*.txt) and 'logs' the first line in the file as well as the number of 'e' in the first line:
string fileName=ParseFilePath(3, filePath, ":", 0, 0)
// ----- parse something -----
int fileID, eCount=0
string read=""
Open/Z/R fileID as filePath
if (V_flag)
return $"" // failed to open => return nothing
endif
FReadLine fileID, read // read the first line
Close fileID
string parse = read, e_Str
do // count the number of 'e' in first line
SplitString/E=("([eE]+)(.*)") parse, e_Str, parse
eCount += strlen(e_Str)
while(strlen(read))
e_Str = num2str(eCount)
// ----- write output -----
Make/Free/T/N=(4,2) info // 2-col text wave
info[][0] = {"tag", "first line", "no. of 'e'", "comments"} // column titles
info[0][1] = fileName // tag identifying the data (could be anything)
info[1][1] = read // content of the first line
info[2][1] = e_Str // no of 'e' in first line
info[3][1] = "" // always empty => entered by the user later
return info
end
Note that you could potentially do other things in this function as well, since your function is called every time a file appears in the folder as long as the watchdog is running. For example, this function could at the same time provide post-processing functionality for your data.
Zip Support
The logger also supports zipped data (with Igor versions 9 and above only). For this, add 'zip' under File Types. The logger will then unpack any zip file and checks the contents for the right extension. Note that the default extension is *.ibw if you do not specify any additional extension. If instead you want the logger to look for 'dat' files within a zip file, enter 'zip;dat;' under File Types.
NOTES:
- Any folder structure within the zip file is currently not supported. While the full zip file is unpacked each time only data at the top level will be processed.
- Unzipping may take a while depending on the file size. Thus, the logger may seem to 'hang' during this process.
- Zip files are unpacked into the temporary folder of the OS. However, only logged files are deleted afterwards. If the zip file contains a lot of unrelated files, this may inflate the size of your temporary folder over time.
Quick-logging
If you have already folders of data and just want to quickly generate a data log from that folder without opening up the user interface you can use the Folder Quick-log function from the Spectra Tools menu. After invoking the menu a file selection dialog will open up. Navigate to the desired data folder and select one of the files of the type you would like to log. This selection automatically decides the file extension to be logged. The quick logger will then create a log file in the same folder. If you want to further edit such a log file, open up the user interface and select the log under 'Save Log File to'. Then press 'Read Log Entries' and start editing right away.
Important Note
Note that this package writes log files to your hard drive, may unpack data to and delete files from the temporary folder of the OS, so here is the obvious disclaimer. Use this software at your own risk. No warranty is offered against error, data loss or any other damage.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
Project Details
Current Project Release
| Release File: | WatchDog_v1.03.zip |
| Version: | IGOR.8.00.x-1.03 |
| Version Date: | Fri, 11/24/2023 - 02:15 am |
| Version Major: | 1 |
| Version Patch Level: | 03 |
| OS Compatibility: | Windows Mac-Intel |
| Release Notes: |
|
Forum
Support
Gallery
Igor Pro 9
Learn More
Igor XOP Toolkit
Learn More
Igor NIDAQ Tools MX
Learn More
I'll attach the example text parser here for reference. I am also happy to help with writing a parser for your specific data. Please also let me know about any bug or feature requests.
July 27, 2023 at 11:45 pm - Permalink