File Input/Output

Making videos with FFmpeg

KZarzana — Thu, 28 Sep 2023 23:15:21 +0000

Making videos with FFmpeg Windows Mac-Intel KZarzana Thu, 09/28/2023 - 04:15 pm File Input/Output Graphing Visualization/Gizmo

The videos I've made with the built in Igor commands never turned out great, but I've had good luck using Igor to make the frames and then calling a second program called FFmpeg to convert the frames into a movie. This ipf creates a user interface for calling FFmpeg through Igor. It only gives the user access to a few of the many features of FFmpeg, but it has worked great for the types of videos that I have made (a few minutes long and embedded in Powerpoint or uploaded to Youtube). The zip file contains the ipf, an example ipf for making frames, as well as the read me pdf.

Supported Igor Version 9.00.x 8.00.x

Spectra WatchDog - On-the-fly and interactive data logger

chozo — Fri, 28 Jul 2023 06:36:12 +0000

Spectra WatchDog - On-the-fly and interactive data logger Windows Mac-Intel chozo Thu, 07/27/2023 - 11:36 pm Data Acquisition File Input/Output Utilities

If you want to create data logs while (or after) measuring data with the ability to edit entries on the fly, then this package is for you.

Spectra WatchDog watches a specified folder for new data files of a specified extension, and then records specified metadata in a log entry. This is most useful if your measurement software dumps finished files in a single folder, which will then be automatically processed. Alternatively, a log can be created from a folder of prerecorded data to generate an annotated summary. The log can be edited (e.g., adding notes or changing entries) in an user interface within Igor Pro. The parsing function is provided by the user (see below). Currently, a function for loading Scienta-Omicron SES files (*.ibw format) is provided with the package. Support for zipped data is provided as well (the data will be temporarily unzipped in this case). Log files are simple tab-delimited text files with the ending *.log (the ending can be changed in desired).

I tried my best to test everything thoroughly, but sometimes bugs slip through. Bug reports and suggestions for new features are always welcome. I am happy to add definitions for popular data formats and open to feature requests.

The User Interface

After installing the package two new options appear under the menu entry Spectra Tools: Folder Watchdog and Folder Quick-log; the latter will be explained later. After selecting Folder Watchdog the user interface appears:

First, there are a few things to set up:

Folder to Watch: Specify the folder to be watched for new data files or to read data from.
Save Log File to: Specify where the log is written to. This file can be saved anywhere you have write permission.
- Press Read Log Entries if you want to read the entries from the specified (existing) log into the table below. This is useful if you continue working with an existing log file.
File Type(s): Specify the file ending(s) to watch for. You need to provide at least one loading function for the specified file ending (see below). Multiple file types can be specified as a semicolon separated list.
- Set Ignore Existing Files: This will ignore all currently available files in the folder and only checks for newly added files.

Now you can press either:

Start Watchdog: The program logs all currently available files with the correct ending and then keeps watching for new files. If you want to stop the process, press Stop Watchdog.
Read Folder Once: The program just logs all currently available files.

The remaining buttons are:

Reset All: Clears the table and resets the list of already processed files in the folder. Use this to start anew. You might want to specify a new log as well. If you don't specify a new log file, then files already logged in the file will not be reprocessed, since you cannot have the same data entry in the log twice.
Clear Display List: Clears all entries in the table below (NOT in the log file). Use this if the table got very long and you want to concentrate only on new entries.

NOTE: Any setting you make will be saved in a file named 'Spectra WatchDog settings.dat' next to the procedure file. This is a simple text file, which can also be edited with a text editor.

Here is how above log file looks in Excel:

CAUTION: Do not open the log file in Excel (or any other program that locks write access to the file) at the same time while you are still working with it in the Watchdog. Igor cannot update the log in that case.

Working with the Table

Logged entries can be directly edited in the table. Simply click a cell to edit it's contents. Editing any cell with also update the log file on the fly. In other words, the contents of the log and the table are synced. The cell in the first column is highlighted for each selected row. The first column is meant to serve as a data tag for identifying the data entry.

Cell selection:
Select single cell with the mouse. Holding shift or cmd / ctrl will select multiple consecutive or disjointed cells respectively. ctrl/cmd+a will select everything (be careful not to delete everything in this case).

Column width and color:
Drag the divider lines in the column header to adjust the cell width of this column. Right-clicking on the cell header allows you to select a color for the column. Both settings will be remembered.

Column options:
In the same, right-click menu of the cell header are additional options:

Widths:

Default: The default width of the package (optimized for the included file logger).
Optimal: Tries to adjust the columns to allow for optimal spacing depending on the current contents in each column.
Equal: All widths will be the same.

Repeat Values option:
If a new entry is written, the content of the previous entry will be prefilled for this column. Use this setting for columns which contain mostly the same information which you do not want to insert repeatedly.

Cell options:
Right-clicking on a cell when not in edit mode, reveals some common options:

Keyboard shortcuts:

cmd/ctrl + a = select all
cmd/ctrl + x = cut
cmd/ctrl + c = copy
cmd/ctrl + v = paste
cmd/ctrl + z = undo/redo
del key = delete
cursor keys = move selection

Writing a Parsing Function

The most important part for using this package with your own data is to write a function with parses the meta data from a file. You can parse as few contents as you like, but the more information is automatically gathered from the files the easier is the logging process on the user.

The parsing function need to have the following format:

function/wave WDfunc_EXT_myFunc(string filePath)

    ...

    return textWave // two columns required: [][0] = entry name, [][1] = value

end

Thus, the function name needs to start with 'WDfunc_' to be recognized by the tool, followed by some EXTension: This must be the file extension, such as 'txt', 'ibw', 'tiff', 'dat' etc. you are going to support with this function. You can have multiple functions for the same extension, if you have different types of files which share the same extension or you can process all these different types in a single function. Finally, the function ends with a freely choosable name.

The filePath string is the full path to one file on disk which should be parsed. The task of this function is to give back a two-column text wave with the desired metadata parsed from the file (and some extra rows for manual information if desired). This can be a free wave. The first column of this wave is the title of the information, which will provide the table's column titles. The second column is the information itself. If your parser encounters the wrong file return an invalid wave ($"") to indicate that this file is not suitable and should be ignored. The logger will then attempt to use another function for this extension (if available) until the file could either successfully be parsed or the file is ignored for good.

As an example, here is a parser which will read text files (*.txt) and 'logs' the first line in the file as well as the number of 'e' in the first line:

function/wave WDfunc_txt_myTextParser(string filePath)

    string fileName=ParseFilePath(3, filePath, ":", 0, 0)

    // ----- parse something -----

    int fileID, eCount=0

    string read=""

    Open/Z/R fileID as filePath

    if (V_flag)

        return $""  // failed to open => return nothing

    endif

    FReadLine fileID, read  // read the first line

    Close fileID

    string parse = read, e_Str

    do              // count the number of 'e' in first line

        SplitString/E=("([eE]+)(.*)") parse, e_Str, parse

        eCount += strlen(e_Str)

    while(strlen(read))

    e_Str = num2str(eCount)

    // ----- write output -----

    Make/Free/T/N=(4,2) info    // 2-col text wave

    info[][0]  = {"tag", "first line", "no. of 'e'", "comments"} // column titles

    info[0][1] = fileName       // tag identifying the data (could be anything)

    info[1][1] = read       // content of the first line

    info[2][1] = e_Str      // no of 'e' in first line

    info[3][1] = ""         // always empty => entered by the user later

    return info

end

Note that you could potentially do other things in this function as well, since your function is called every time a file appears in the folder as long as the watchdog is running. For example, this function could at the same time provide post-processing functionality for your data.

Zip Support

The logger also supports zipped data (with Igor versions 9 and above only). For this, add 'zip' under File Types. The logger will then unpack any zip file and checks the contents for the right extension. Note that the default extension is *.ibw if you do not specify any additional extension. If instead you want the logger to look for 'dat' files within a zip file, enter 'zip;dat;' under File Types.

NOTES:

Any folder structure within the zip file is currently not supported. While the full zip file is unpacked each time only data at the top level will be processed.
Unzipping may take a while depending on the file size. Thus, the logger may seem to 'hang' during this process.
Zip files are unpacked into the temporary folder of the OS. However, only logged files are deleted afterwards. If the zip file contains a lot of unrelated files, this may inflate the size of your temporary folder over time.

Quick-logging

If you have already folders of data and just want to quickly generate a data log from that folder without opening up the user interface you can use the Folder Quick-log function from the Spectra Tools menu. After invoking the menu a file selection dialog will open up. Navigate to the desired data folder and select one of the files of the type you would like to log. This selection automatically decides the file extension to be logged. The quick logger will then create a log file in the same folder. If you want to further edit such a log file, open up the user interface and select the log under 'Save Log File to'. Then press 'Read Log Entries' and start editing right away.

Important Note

Note that this package writes log files to your hard drive, may unpack data to and delete files from the temporary folder of the OS, so here is the obvious disclaimer. Use this software at your own risk. No warranty is offered against error, data loss or any other damage.

THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

Supported Igor Version 9.00.x 8.00.x

chozo

I'll attach the example text parser here for reference. I am also happy to help with writing a parser for your specific data. Please also let me know about any bug or feature requests.

Attachments Attachment Text Parser.ipf

July 27, 2023 at 11:45 pm - Permalink

VASP Loader

Yuan Fang — Sun, 09 Jan 2022 04:20:49 +0000

VASP Loader Mac-Intel Yuan Fang Sat, 01/08/2022 - 08:20 pm File Input/Output

Read bands generated by VASP, and can create projected bulk bands to compare with ARPES bulk measurements. Also, it can show orbital weight contained in PROCAR. DOS can also be read and plotted through the panel.

Supported Igor Version 8.00.x

ExifTools

KZarzana — Fri, 22 Oct 2021 20:48:01 +0000

ExifTools Windows Mac-Intel KZarzana Fri, 10/22/2021 - 01:48 pm File Input/Output Graphing

A short set of functions to read and write exif data on image files. Used to keep track of information related to when and where the image was generated. Stores the path and name of the experiment, the date and time the image was created, and the graph name and title. Requires the installation of the free ExifTool utility (https://exiftool.org/).

Supported Igor Version 8.00.x

netCDF utilities

cpr — Thu, 01 Apr 2021 13:40:55 +0000

netCDF utilities Windows Mac-Intel cpr Thu, 04/01/2021 - 06:40 am File Input/Output Utilities

New features in Igor 9 now allow the correct reading and writing of netCDF-4 files directly from within Igor. Igor 8 already added native netCDF-3 loading ability and Igor 7 added support for the extended data-types of netCDF-4 e.g. uint64 etc. Here are two sets of utilities to do both reading and writing using all in-built operations and user defined functions.

QLA netCDF reader

Loads netCDF-3 and netCDF-4 files such that the original type is transparent to the user
Parse all netCDF metadata nicely into CDL syntax attached to the WaveNote (c.f. ncdump or Panoply) or text dump
Set the units and dimension names of the dataset
Optionally apply scale factor, offset, fill value, valid range to data
Functions to list, get, or set the attributes of a dataset
Function to determine the type of netCDF file
Menu is provided for convenience, inline is also perfectly fine

QLA netCDF4 library

Functions to create netCDF-4 or netCDF-4 'classic model' files directly from Igor
Switch between types easily without changing any code
Keeps track of dimension information
Keeps track of _Netcdf4Dimid and _Netcdf4Coordinates required hidden attributes
Keeps track of unlimited dimensions, layout, and chunking
Keeps track of the hierarchy, duplicate names, dimensions size matching etc.
Provides some basic help for keeping CF compliance
Provides some hopefully helpful error/information messages in the history area
Compression and shuffling is available
Functions follow similar syntax to netCDF C library functions
Functions vaguely named after netCDF functions e.g. NC_def_var becomes QLA_def_var

A sample .ipf is included which uses almost all the functions available that you can use to make your own netCDF-4 / netCDF-4 classic and figure out how it all works, test the files using ncdump, nccopy, or the included loader etc.

Limitations:

Only HDF5 based netCDF-4 files can be created directly. To create a netCDF-3 use the netCDF API nccopy.exe to convert your netCDF-4 classic model into a netCDF-3 i.e.

// convert a netCDF-4 classic into a netCDF-3

ExecuteScriptText /B /Z "cmd.exe /C nccopy -3 myNC4classic.nc myNC3.nc

More exotic datatypes like ENUM are probably possible but aren't currently included in the library

All credit goes to Howard Rodstein for all his effort with changes and additions to the HDF5 operations within Igor. These include but are probably not limited to:

New HDF5DimensionScale operation - this is what makes creating netCDF-4 dimensions possible
HDF5 attribute creation order tracking and indexing
Fixed length HDF5 string attributes
Loading of VLEN type data needed for reading netCDF-4 dimensions
HDF5SaveData /STRF flag for explicit control of text formatting
HDF5SaveData /OPTS flag for controlling the type of dataspace; required for scalar variables

Supported Igor Version 9.00.x

LEEM-PEEM Analysis

F. Cheynis — Fri, 11 Dec 2020 09:20:43 +0000

LEEM-PEEM Analysis Windows Mac-Intel F. Cheynis Fri, 12/11/2020 - 01:20 am Analysis File Input/Output Graphing Image Processing User Interface Utilities

About

LEEM-PEEM Analysis a program package to extend the image analysis tools provided by Elmitec GmbH acquisition software Uview2000 for the Low-Energy Electron Microscopy (LEEM) and Photo-Emission Electron Microscopy (PEEM) techniques. This package is freely available to the LEEM-PEEM user community considering that the authors will be pleased that the following reference be cited: Rev. Sci. Instru. 85 (2014) 043705.

Features

The Graphical User Inferface (GUI) is divided into three parts. The most outfitted component concerns Elmitec GmbH Uview2000 movies (.dav) or image sequences (.dat sequence). These files are opened with a dedicated button. Vizualisation of the sequence is then possible. The built-in "Image Contrast" panel developed by Wavemetrics can be used to adjust the image rendering.

Standard file-handling procedures are implemented such as Extract (e.g. to select certain images of the original data and create a new movie) and Convert (to generate a .mov or a .avi movie for presentations).

Image processing includes polynomial background removal (Backgrnd) using Igor Pro built-in procedure.

Two approaches are implemented for image drift correction (Drift. Corr.). The first is based on an automated image-to-image intercorrelation. The second approach requires that the user marks/locates a given detail on the surface under consideration in different images of the sequence. The drift is then linearly interpolated between these positions and a new movie is created.

A particle analysis procedure (Part. Anal.) is also available. An arbitrary number of particles (i.e. black, resp. white, areas on a white, resp. black, background) are characterized (area and x, y position) as a function of the movie image number. Manual and Iterated built-in thresholding methods are implemented to detect automatically the current position of a given particle. A procedure to measure the intensity of a given region-of-interest along the image sequence is implemented (I_ROI). This is particularly useful for reflectivity I(V) curves.

The second and third components of the GUI enable to display in Igor Pro .mov movies and various types of individual images obtained from Elmitec acquisition software (.dat) or in standard formats (.tiff, .png, and .jpeg).

Install

You need Igor Pro 6.3 or higher.

Unpack the .zip file.

LEEM_Analysis2009.ipf and LEEM_Analysis2015.ipf files are global procedures. So simply move them to the 'Igor Procedures' folder or in a folder with an alias pointing to it in the 'Igor Procedures' folder.

All other '.ipf' files are shared procedures. Move these '.ipf' files in the 'User Procedures' folder or in a folder with an alias pointing to in in the 'User Procedures' folder.

Start Igor Pro. 'LEEM analysis 2009' and 'LEEM analysis 2015' items should be available in the 'Macros' menu. Choose the item according to the year release of your Uview2000 software. This creates the main panel. Enjoy !!

Limitations

So far only 1024x1024 images and image sequences are supported.
This package is provided as-is without any warranty.

Supported Igor Version 6.30.x

X-Y Browser

tony — Thu, 04 Jun 2020 15:48:19 +0000

X-Y Browser Windows Mac-Intel tony Thu, 06/04/2020 - 08:48 am File Input/Output Graphing User Interface

X-Y Browser is a user interface for previewing Y vs X plots of paired waves. Waves are paired by specifying the required X wave in the wavenote of the Y wave as a key pair. The GUI can also be used to preview X-Y plots of specified columns of 2D waves.

The project adds menu items to the Data and Windows menus. The Paired Wave Browser panel allows the user to select wave pairs or 2D waves and to preview them in a Y vs X plot.

The browser is useful for building Y vs X plots of many pairs of waves, where the waves cover different spectral ranges. One reason to use X-Y wave pairs rather than 2D waves is to maintain compatibility with various tools for spectroscopy that will not work with 2D waves.

This is designed to be combined with a custom file loader that sets the wavenote. An example is the WiRE loader for Renishaw Raman files. It should not be used for spectra that have constant spacing of points! Using the SetScale operation to set the x-scaling for constantly-spaced data will unleash a world of functionality, and is one of the things that helps make Igor Pro such an indispensible tool for spectroscopy. If, however, you are forced to work with X-Y data, X-Y Browser may make your life a little easier.

Supported Igor Version 9.00.x 8.00.x

WiRE file loader for wdf files

tony — Tue, 02 Jun 2020 15:56:10 +0000

WiRE file loader for wdf files Windows Mac-Intel tony Tue, 06/02/2020 - 08:56 am File Input/Output Graphing

A file loader for wdf files created by Renishaw's WiRE software for Raman spectroscopy.

Line scans and map grids are imported as multiple 1D waves. The corresponding Raman shifts are saved in an accompanying wave. The name of the wave containing the Raman shift is recorded in the wavenote for each data wave. This allows pairs of waves to be browsed and plotted efficiently using a custom X-Y browser (I will make a separate project release for something like that). Embedded images are extracted from the wdf file and can be plotted with an overlay of map grid points. Mouseover grid points in the map browser GUI to display spectra and click to select spectra for plotting.

The project adds an item to the Load Waves menu, and the map browser can be found in the Packages submenu of the Data menu.

Note that this loader is based on some guesswork about the file structure, and I haven't tested it extensively.

Supported Igor Version 8.00.x

JSON XOP

thomas_braun — Mon, 15 Jul 2019 18:45:14 +0000

JSON XOP Windows Mac-Intel thomas_braun Mon, 07/15/2019 - 11:45 am Packages File Input/Output Programming Utilities

JSON is a format for exchanging data between applications. This XOP implements a 100% standards compliant interface for reading and writing arbitrary JSON files.

Documentation and Downloads: https://docs.byte-physics.de/json-xop/

The XOP is available for Igor Pro 8 free of charge.

Supported Igor Version 8.00.x View Homepage

ukos

I was looking for something like this. This is really helpful. Thank you.

July 15, 2019 at 02:48 pm - Permalink

SPEC Loader

hrodstein — Tue, 04 Jun 2019 16:48:09 +0000

SPEC Loaderhrodstein Tue, 06/04/2019 - 09:48 am File Input/Output

This code loads the data from a SPEC file. It requires Igor Pro 7.00 or later.

From http://www.certif.com/content/spec/

spec is a UNIX-based software package for instrument control and data acquisition widely used for X-ray diffraction at synchrotrons around the world and in university, national and industrial laboratories.

The code adds a "Load SPEC File" item to the Data->Load Waves menu.

See the comments at the top of the "SPEC Loader.ipf" file for further information.

Supported Igor Version 7.00.x