Missing path in experiment file (macOS)

When you open a procedure file using the open file dialog or by clicking on the file, if the file is moved from that path*, even if it's in user procedures, Igor will not know where to find it.

In contrast, if you use #include to load a file from user procedures it does not matter if the username or path to User Files changes.

It's not much help in your current situation, but maybe a good idea to use #include if absolute paths** are likely to change.

--edit:

* see hrodstein's more detailed explanation below

** actually, if both absolute AND relative paths are changed

Ah of course! That makes sense. I do use #include but generally for very stable bits of code I've written. These procedures are shared between fewer experiments, so I don't currently use #include. I see that I need to change the way I'm working now that I've run into this. Thank you!

Here are my 2 cents: I think there is no reason to not use #include also (and especially) for not-so-stable-and-finshed code. I mean, the code is not loaded in general and only if you really want. That's precisely what #include is for, and it will make sure the files are loaded from Users Procedures. If the code is only meant for one specific experiment file then it may as well live in the main procedure window, which is always saved with the experiment file and does not rely on external files. Now, to solve the dilemma with your current files, you could adopt the procedures on your original Mac (File => Adopt Procedure), which makes the procedure local to the experiment and independent of the external file. The drawback is that this code then of course doesn't get updated anymore.

As a side note, if the code is stable enough and used all the time, I go ahead put them into the Igor Procedures folder to load them all the time without any #include statement. This prevents experiment files from breaking when I send them to users who don't have my files (assuming you make sure that nothing breaks otherwise if the code is not loaded).

Thanks, chozo. I appreciate your comments. Re: loading stable code in Igor Procedures, is the overhead negligible for doing that? I mean, is it best to limit the files accessed this way? I am thinking that several routines only become available in Igor upon selection, e.g. ImageTools. Is that to keep things uncluttered or is there a performance benefit to not auto-loading too much stuff?

How experienced users manage their WaveMetrics folder for their Igor Pro installations may be of general interest, especially to newer users. In this respect, as a modestly experienced user, I offer a few comments about my practices.

* I have only the stand-alone procedures below in Igor Procedures. They are there because I use them consistently. I install all other stand-alone procedures as needed (more below). 

- Procedure Loader https://www.wavemetrics.com/node/21127
- Project Updater https://www.wavemetrics.com/project/Updater
- Window Notes https://www.wavemetrics.com/node/21657
- LogBook https://www.wavemetrics.com/project/LogBook

* I also have a procedure file in Igor Procedures. The procedure generates menu options for the Macros menu.

The selections are actions that I commonly execute depending on the experiment. For example, as I develop a package, I almost always have to work in independentmoduledev=1 mode. Alternatively, as I am analyzing data, I often want to include a time stamp in the history window to mark my progress or branch the current experiment to a snapshot (e.g. for archival storage).

To answer questions about overhead ... I see no noticeable issues with lag from overhead when Igor Pro starts up and loads the procedures in Igor Procedures.

* In the past, I used Igor primarily for analysis in spectroscopy. At that point, I also included an analysis package (SpXZeigR) in my Igor Procedures folder. Fast forward to today. I now use Igor Pro to handle a range of needs from development to data or image analysis. I don't want the overhead from the variations in procedure files to clutter my start up with Igor Pro. To manage this, I keep (a whole lot of) procedures in User Procedures and install only the ones that I need as I need them for each experiment. For this aspect of working in Igor Pro, I highly recommend Tony's Procedure Loader package (noted in the list above). I no longer take time to open the macro procedure window and type in an '#include ...' directive to add the package that I need to an experiment. I use the File->Load or Unload Procedures menu option that Tony's package affords to handle all such needs.

* Suppose that you want to share an Igor Pro experiment (.pxp file) with a user who may not have all the required procedure files on their machine. What are your options when you would distribute the experiment? One option is to include instructions on what procedures must be installed *before* the experiment can be opened by the other user. The other option is to use the File->Adopt All menu. This option is available when you press the SHIFT key as you access the File menu. Once you save the experiment using the Adopt All method, you can send the experiment (e.g. in a ZIP archive) to your colleague with the assurance that, upon opening, all the components you need are included in the experiment itself.

* Finally, a core aspect to managing procedure files is keeping them updated. Various developers take different approaches to how they post updates for procedures. When a package is posted on Igor Exchange, I highly recommend making use of Tony's Updater package (also noted in the above list). Coming from a long history with LaTeX and its TeXLive installer, I find Tony's Updater package adds a comparable professional-level approach to how developers can post their projects and how users can keep their installations of those packages updated.

Many thanks jjweimer.

I can offer my previously described method to deploy code to a team and keep it updated without them needing to know too much about the details. https://www.wavemetrics.com/forum/general/solution-sharing-and-updating… 

@sjr51 -- This also adds to the recent discussions at Tony's Project Updater page. The continuing question seems to be how a developer of Igor Pro procedures can best distribute the procedure files to a team when he/she does not want to use Igor Exchange. We still seem to have a hodgepodge set of approaches.

I have to wonder what effort would be involved to provide an Igor Pro package equivalent to Tony's Updater package that can be pointed to a user-defined cloud service location to pull package files and install them locally. The idea for example is that I would develop an update to my procedure file XYZ.ipf, post the update to my shared cloud service, and inform my team that an update is posted. The team would use the cloud updater package to pull down and install the update locally (e.g. into their User Procedures folder). If I understand your solution, the caveat is that the procedure files in your UP cloud folder are run from that cloud folder, not downloaded and installed locally before being run.

The procedure loader will follow shortcuts in the User Procedures folder. However, I just tested this with a networked folder and the performance was inadequate. This is because the procedure loader makes a recursive directory listing and examines the files for compatibility when it creates a list of procedures. It looks like sjr51's solution works with a local cache of the networked directory listing.

I keep about 100 procedure files in my User Procedures folder, and my personal settings #include 16 of these by default.

As noted by others, you can avoid this kind of problem by using putting your procedure file in "User Procedures" or "Igor Procedures" and using the appropriate #include syntax.

For those rare situation where #include is not desirable, everything will work if use a packed procedure file or you keep the relative location of the target procedure file and the experiment file unchanged. But using #include is less fragile.

You quoted the path as "::::WaveMetrics:Igor Pro 9:User Files:User Procedures".

The normal layout of Igor Pro User Files folder is "Igor Pro 9 User Files:User Procedures", not "Igor Pro 9:User Files:User Procedures", and I will assume that the former is your actual path. (The two paths differ by the character after the "9": space in the former, colon in the latter.)

Igor is trying to locate the folder associated with a symbolic path in the experiment being loaded.

"::::" introduces a relative path and means "up three folder levels relative to the location of the experiment folder". (":Folder" means "in Folder", "::Folder" means "up one level from Folder", ":::Folder" means "up two levels from Folder", and so on.)

Here is an example of a folder hierarchy that satisfies this path:

The path is relative to the experiment folder which is SubFolder3. "::::" means go up three levels so that gets us to Documents. The rest of the path says go into "WaveMetrics" then go into "Igor Pro 9 User Files" then find the folder "User Procedures".

If you move Test.pxp somewhere else, the relative path will not work. In that case, Igor attempts to locate the folder using a full path which is stored in the experiment file to help resolve situations like this. The full path might be something like "hd:Users:<user>:Documents:WaveMetrics:Igor Pro 9 User Files:User Procedures". If the user name (denoted as <user> in this path) is changed then the full path will not work.

So you will get the missing folder dialog if the relative path does not work and if the original full path does not work.

The relative path does not rely on the user name. I suspect it is not working because your arrangement of files has changed. One way to fix this is to move the experiment file so that the relative path works.

I experimented with this situation and was also able to fix it by creating a symbolic link like this:

This creates a soft link from the old location of "Igor Pro 9 User Files" to its new location.

The folder /Users/OldUser/Documents/WaveMetrics already existed. You would have to create it if it does not exist.

This works because it allows the full path stored in the experiment file for the symbolic path to work.

@jjweimer That's correct, the end user just has read-only access to that folder and is just running things from there. If I push an update then the user gets that when they open the experiment and can then rerun the code or whatever.

I'm using GitHub for maintaining some Igor code and distributing to the world (should they want my stuff!). I found that folks in my team were not updating - often because they didn't know there was an update. In honesty, downloading code from GitHub and putting it in the correct folder was beyond some of them!

It would be nice to have a solution for all of this. In R users can do `devtools::install_github("user/nameOfRepo")` which works well for deploying updates; in Fiji/ImageJ there are Update Sites that allow developers to push code very effectively. I am saying all of this yet I haven't tried Tony's updater package - which looks very nifty indeed.

@hrodstein thank you! If only I could click your answer as the accepted solution. Firstly, you are correct that my path had a typo - sorry. Secondly, your fix of putting the pxp back in the correct location does indeed fix the problem regardless of username on the filesystem.

To expand on this. I was pulling down a pxp from my server to my Desktop on my new Mac and, when running it from there, it gave the error (which I assumed was due to the username change). The directory structure of the server is mirrored on the Mac and so if I place the file at the equivalent location on my Mac, the pxp opens and the procedure file is found in User Procedures just fine. By experimenting, the file needs to be at the correct depth. If it's in a subfolder or parent folder of where it should be, I get the missing path dialog.

It makes sense now why I haven't run into this previously. I'm pretty sure with my previous username, I could pull down pxps and run them from wherever I wanted and the paths just resolved (because the symbolic link was correct). I guess with a username conflict, it looks for relative paths as you say and I had messed that up by running from the wrong depth in filesystem.

Thank you also for the soft link solution. It's great to know that this works too.

Other folks who replied to me telling me to use #include and other helpful tips can all have a virtual upvote.

jjweimer wrote:

I have to wonder what effort would be involved to provide an Igor Pro package equivalent to Tony's Updater package that can be pointed to a user-defined cloud service location to pull package files and install them locally.

One way is to use a version control system that's built for the job.

If you have files on a networked drive that you want to keep synced with a local folder, some kind of third party synchronization software or OS capability could fit the bill. Alternatively, synchronization can be done from within Igor. I have an Igor function for syncing folders, and simply firing that from a startup hook or background task might be sufficient.

code snippet

I'm pretty sure with my previous username, I could pull down pxps and run them from wherever I wanted and the paths just resolved

It worked because Igor stores, in the Igor experiment file, the full path to the folder for a given symbolic path. If the relative path does  not work, Igor tries the full path. The full path worked before but stopped working because your user name changed.