This week, guest blogger Peter Webb continues his series of articles about the MATLAB Compiler. This week's topic: managing MATLAB's paths in a compiled application. For an introduction to writing deployable code, please see the June 19th article.
Contents
One of the goals of the compilation process is to turn a flexible and easily modifiable MATLAB progam into a robust software component with consistiently predictable behavior: a compiled application should not be able to change which functions it calls or the way those functions work. As a result, certain aspects of the execution environment that are malleable in MATLAB become fixed or constant in a deployed application. Perhaps the most important of these is the mechanism by which MATLAB locates functions and data files: MATLAB's search paths.
A path consists of a list of directories that MATLAB searches to find files. MATLAB typically searches a path in list order, stopping at the first file that matches the search criteria. MATLAB uses paths for two reasons: to determine which functions to execute; and to locate data files in the file system.
MATLAB primarily interacts with four paths, two internal and two external: the MATLAB path; the MATLAB Java class path; the system path; and the system load library path.
Most familar, perhaps, is the MATLAB path, which affects how MATLAB determines which MATLAB function files to run and which MAT-files to open. Though the MATLAB path is the most visible, of nearly equal importance is the MATLAB Java class path, which specifies the directories in which MATLAB searches for Java functions. The two external paths list the locations which the operating system will look for executables and shared libraries, respectively. The Windows operating system uses the same path to look for executables and shared libraries, but most Unix systems search separate paths for these two types of files.
When writing an application intended for deployment via the MATLAB Compiler, keep in mind that MATLAB applications typically interact with paths in at least three independent ways:
- Executing MATLAB commands to change a path
- Passing paths as arguments to MATLAB functions
- Relying on the existence or structure of external paths
After briefly listing some general guidelines for safely interacting with paths in MATLAB, I provide more detailed techniques to manage problems that can occur in each of these areas.
General Guidelines
A compiled application behaves more consistently if it:
- Does not change any path during execution.
- Uses relative paths (or anchors paths to a known root via the matlabroot or ctfroot functions).
- Avoids accessing or changing the current directory.
These guidelines become more important when the compiled application executes in an envrionment that differs from the one in which it was developed: for example, when installed at a customer site.
In cases where it is convenient to keep path management commands in your M-files when they run in MATLAB, you can use the isdeployed function to skip over calls to path management functions in deployed applications. For example, this M-code adds my "beta"-quality M-files to the path when my applications run in MATLAB:
if ~isdeployed addpath /home/pwebb/mfiles/beta end
MATLAB Commands that Directly Access the Path
All M-file and Java functions rely on the state of MATLAB's internal paths, but only a few MATLAB commands directly access or modify these paths. Most of these functions should be avoided in deployed applications because they have the potential to change the way the application works, either by changing the functions it calls or the data files it loads.
| addpath: | Add a directory to the MATLAB search path. |
| cd: | Change the current directory. |
| javaaddpath: | Add a directory to MATLAB's Java search path. |
| javaclasspath: | Get or set MATLAB's Java search path. |
| javarmpath: | Remove a directory from MATLAB's Java search path. |
| path: | Get or set the MATLAB search path. |
| rmpath: | Remove a directory from MATLAB's search path. |
| savepath: | Save the current MATLAB path. |
Path management commands often appear in startup or initialization code; when it comes time to deploy your application, you may have forgotten that the application relies on these path settings. Problematic path settings sometimes occur in these oft-forgotten files:
| pathdef: | Used by MATLAB to establish the initial MATLAB search path, pathdef.m may be modified, in particular by the savepath command. |
| startup: | The MATLAB Compiler incorporates startup.m into every generated executable and shared library; any path management commands in startup.m therefore need to work in the deployed application. Directories added to MATLAB path by startup.m are automatically added to the MATLAB path in a deployed application by the MATLAB Compiler if the deployed application uses any files from those directories, thus making the addpath calls in startup.m both redundant and dangerous. |
Protect all calls to path management functions in these files by enclosing them in an ~isdeployed if-block.
Instead: Ask yourself why you're changing the path. Rename functions or use MATLAB Objects to manage your function namespace. Change the path before running the MATLAB Compiler to add optional or conditional functionality to your application.
Instead: Use paths relative to matlabroot or ctfroot.
Change code like this:
cd my/data/directory fp = fopen('data.file', 'r');
To code like this:
fp = fopen(fullfile(ctfroot, 'my', 'data', 'directory', 'data.file'));
Instead: Set the Java class path before running the MATLAB Compiler. Use classpath.txt or javaaddpath.
Instead: If you must save the path, save it to a MAT-file; otherwise, enclose the call to savepath in an ~isdeployed if-block.
p = path; save savedpath p
As further incentive to avoid these functions in deployed applications, please note the commands that change the Java class path (javaaddpath, javarmpath) also clear the values of global variables.
For more details on how to manage the MATLAB function namespace with MATLAB objects, please see the polynomial example in the MATLAB documentation.
Paths as Arguments to MATLAB Functions
Many MATLAB programs assume that the directory structure of the filesystem cannot change. When this assumption leads to the use of absolute paths in file names, e.g., load('c:/Work/MATLAB/data.mat') or an implicit dependency on MATLAB's ability to locate a file, e.g., load data.mat, it creates problems for deployed applications. A deployed application is very likely going to be installed on a machine that has neither the same directory structure nor the access to the same filesystem as the machine on which it was developed. The use of absolute paths causes errors because of differences between the development and the deployment file system structures. But relying entirely on relative paths establishes a fragile dependency on the location of the current directory. The solution: base all file locations on a known root.
In MATLAB, there are typically two file system roots that matter: the root of the external file system, typically / or c:/ on Unix and Windows systems respectively; and the MATLAB root, the directory in which MATLAB has been installed. In a deployed application there is an additional root: the CTF root, which is the location of the application's MATLAB content (M-files, data files, JAR files -- everything that the MATLAB Compiler put into the application's CTF file). In MATLAB ctfroot and matlabroot refer to the same directory: MATLAB's installation directory. In a deployed application, matlabroot refers to the directory in which the MATLAB Common Runtime (the MCR) is installed and ctfroot to the CTF root.
Deployed applications typically base their file names on the the CTF root. For example, assume an application that reads and writes MAT files. To ensure that the MATLAB Compiler includes the input data files in the CTF archive, specify the data files with mcc's -a (add) flag:
mcc fcn.m -a data.mat
In the application code, use which or ctfroot to construct a full path to the files being loaded or saved.
Use which when a known filename exists on the application's path:
[pathstr,name]=fileparts(which('data.mat')); % Load x from the MAT file load(name); x = x + 17; % Save x back to the MAT file save(fullfile(pathstr, 'data.mat'), x);
Use ctfroot when creating a new file:
data = input('Enter data: '); save(fullfile(ctfroot, 'more_data.mat'), data, '-append');
This use of ctfroot assumes that the application has permission to write to the file system on the deployment machine. This is not always true by default, especially with network installations. Work with your IT department to secure the appropriate file system access.
Relying on the Structure of External Paths
A MATLAB application that uses the system or loadlibrary functions or calls MEX files linked to non-MathWorks shared libraries has a dependency on the external system paths. The system command allows MATLAB programs to directly execute operating system commands and capture their output as text. This can be very convenient, but it of course requires that the command be available in the current file system. loadlibrary allows MATLAB programs to load shared libraries and call their exported APIs; it similarly requires that the library in question be available on the deployment machine's file system.
The techniques for managing external path dependencies are very similar to those used to manage internal path dependencies: use full paths, favor explicit dependencies over implicit ones, and include all required files in the deployed application.
- Use full path names, rooted in the ctfroot, to specify the arguments to the system and loadlibrary functions.
- Include non-standard applications and shared libraries in the deployed application by using mcc's -a switch.
- Ensure the necessary applications and libraries are present on the deployment machine or in a network location accessible from the deployment machine.
Next Up: Non-supported functions
The next post in this series will explore mechanisms for identifying and managing calls to non-supported functions. In the meantime, you can refer to the documentation for the MATLAB Compiler or post follow-up questions here.
Get
the MATLAB code
Published with MATLAB® 7.6
Hi Loren and Peter,
I have a software package that I currently deploy as a standalone application. I am currently considering releasing some of the matlab code as open source, so that matlab-having clients can develop plugins for my package, but I would like to protect some of the code, for which I can use p-code, but as importantly, I would like to make sure that nobody can compile a standalone application from the open source code I release. The open source permissions for these clients would only extend to within matlab.
The first, obvious solution is to create a critical p-file that checks isdeployed(), and won’t run if it is deployed. The problem I see with this is that isdeployed() can be overshadowed by the user, prior to compilation. Is there a solution to this vulnerability? How can I release truly undeployable .m/.p code? How does The Mathworks protect imtool, etc?
Thanks for a great column, and any help would be greatly appreciated!
Great post Peter!!
The fullfile tip is very useful, I’m sure lots of people will benefit from this post. As a newbie MATLAB programmer, I fell victim to many of the scenarios you listed here. Those changes in path can really be troublesome!
Quan
Compiling data with a deployed application is generally not a good idea. A better approach would be to make the data an input to the application. The path to the data would be supplied by the user of the application either from the command line or using a gui.
Johan,
The MATLAB Compiler doesn’t currently provide any user-accessible ways of preventing the deployment of M-files. We’ve been thinking about this though; it’s a tough problem, which is why you haven’t seen a solution from us yet.
For now, you’ll have to manage this issue yourself. You’ll need to decide exactly how much security you want; the more you want, the more work you’ll need to do. Some ideas you might try:
* Use isdeployed() as you outlined above; yes, it can be overloaded, but how many people will think to do that?
* Use a license manager, like FlexLM, for example, and develop a MEX file to interact with it. Wikipedia has some ideas here: http://en.wikipedia.org/wiki/License_Management.
* Rely entirely on your license agreement with your users, and do not attempt a technical solution.
Keep in mind that there are probably effective attacks on all of these suggestions — there’s no such thing as a perfectly secure system.
Fundamentally, you’re up against a very difficult problem. It’s very hard to simultaneously distribute source code and control how that source code can be used.
Regarding overloading of isdeployed(), can’t you use builtin(‘isdeployed’) as a workaround? (Although I guess builtin could also be overshadowed…)
Hi Peter,
Thanks for bringing up the idea of using ctfroot to make the paths deterministic. However, not only file permissions is a big problem for my project, this also makes it difficult to manage deployed machines when the user have to look under something like
C:\Documents and Settings\Username\Local Settings\Application Data\MathWorks\mcr_cache_v79\ICDviewer_63A7BF5EA103F85B3934BF5A0DE7D64D\data
to find a .mat to send it to the developer for problem tracking!
Is there a way that the MCR can extract the guts (encrypted .m files) according to the same folder structure when it was compiled (i.e. where the .exe is located) and extract the rest (toolboxes and all supporting files) /mcr_cache right under it?
If I were to base my paths off a pivot, it should be the folder where the compiled executable is located on, not some random place on the drive.
If there’s a good rationale for the path management scheme to be like the way it is right now that will benefit the users, please let us know, and I’m sure we can change our habits. On the other hand, if designing it the other way round is difficult, please discuss the difficulties and we’ll mark (comment) our code and wait for the future version.
I also think that a function which returned the path of the deployed exe file would be very useful. As Hoi mentioned above, the return of “ctfroot” is some rather strange path that I find very little use for. I usually solve the problem by creating an installer that sets a system variable to the root directory of the standalone application.
Gunnar: Actually you don’t have to rely on changing the MCR path (although it’s a good idea because the default paths is too long, long enough to break some old DOS program like deltree.exe) and complicate the installation/maintanence process. You can just use setappdata(0, ‘rootFolder’, pwd) at your main program and retrieve the path pivot anywhere.
However, I still don’t like the idea of having MCR messing with the my path structures. This only adds unnecessary mental gymnastics to the programming task, hurting productivity.
If a programming language feature is clearly unintuitive to users like the MCR paths, it better has a good justification other ‘it’s easier to develop the compiler that way’.
Hoi and Gunnar,
You can exercise more control over where the encrypted files are installed via the Compiler’s -C switch or the MCR’s “component cache” environment variables.
The details: http://www.mathworks.com/access/helpdesk/help/toolbox/compiler/brl4_f1-1.html
If you place the CTF archive and the executable in the same directory at installation, then CTFROOT will return the directory containing the executable.
With -C switch, I think ctfroot will be the cleanest solution as the root path. Thanks Peter!
Even with the -C switch, the MCR still breaks my path structure as now /myProgram/core is extracted to /myProgram/myProgram_mcr/core, but it’s better than having it extracted to somewhere deep under ‘local settings’.
To shorten development time, I took a chance on MATLAB compiler when I decided to use MATLAB for a software project meant for distribution, and I found a bunch of things that ought to be right (DCOM, internationalization, single instances, uitable, paths, etc.) that wasn’t. I filed bug/enhancement requests for all of them, but it will take a long while before the developers actually do something about it.
I really hope Mathworks can invest more on developing MATLAB compiler to make it a more polished product. With cheap dual-core computers @1GB RAM minimum ready to run MCR, developing applications (especially GUI with data visualization features) using MATLAB is a competitive alternative to C#/Java.
Hi,
We have some need about a deployement process. We have a application which is compiled by MatLab Compiler last release, V4.8.
When we are in development or in deployement, we need to know the root directory from where the application was launch (localisation of the exe directory).
It’s like CTFROOT but for the EXE.
Do you know if it is possible to have that ?
Thanks a lot.
Best regards.
GALLOU,
I think the information in comment #9 might help you as well. If you compile using -C, and place the CTF archive in the same directory as the executable, then CTFROOT will return the directory containing the executable.
Hoi,
Long-term, we certainly plan on further improving the quality of the MATLAB Compiler (and all of the Builder tools). I’m not sure exactly which bugs you’re referring to; I took a look in our list and didn’t find any mention of uitable, but I noticed several internationalization and COM bug fixes going into the next release, so perhaps we’ve already addressed some of your concerns.
Hi Peter,
I’m glad to hear that you guys have plans to improve the compiler quality. Quality-wise, I think most people can live with minors bugs if there’s a workaround, but if the scheme/idea is not well-thought out (like single instances and default settings like Windows Vista that drives user nuts), it might start to shake people’s confidence about whether they’re using the right tools.
I was tempted to scratch my project and switch to C# for 4~5 times when I had to fight with the MATLAB compiler, and thanks to the quick and helpful support, I got through most of them and was back on track. The most fustrating experience was DCOM: the apps engineer sent me a documentation page that remote DCOM is not supported by the compiler, and I ended up setting up two virtual machines, taking the difference in the Windows+/-MATLAB/MCR installations and found that it’s just one single line that I have to add to get MCR to use remote DCOM! If that’s the one that you’re talking about, I look forward to remove my ‘secret sauce’ in the deployment installer.
For internationalization, if all the developer’s machine are set to display ‘Japanese’ for non-Unicode programs, the daily problems in compiled programs will annoy them enough for them to fix it (even if the developer can read Japanese, the characters are still messed up for half of the times). If the developers assume users set the non-Unicode programs to display Japanese because they can read Japanese (bilingual), then I am hoping that at least if MCR chose to display Japanese error messages in the console, display it properly in Japanese code page. Of course, it’d be the best if English MATLAB consistently shows everything in English, in any OS.
For uitable, an apps engineer called (a long time ago) and promised to work on it, but I hadn’t heard back from anybody ever since. The request ID is 1-5Y36TS. Basically the uitable() in my compiled program displays blank until I refresh it again with something else, and that doesn’t happen to the full MATLAB. We can take this discussion offline for better nettiquette.
I feel MATLAB compiler can expand its territory a lot more. If Steve can have a blog on something as specific as Image processing, I certainly appreciate a blog on Application Deployment branch of MATLAB products, as they can be quite obscure and daunting and maybe a community can help promoting the product through more success cases.
Hi,
I have created a standalone executable (for Windows), called ‘v1′, that has a Help Menu. I want the user to be able to use the Help Menu to open a pdf User Manual, but get an error when trying to do this from the deployed/compiled version. I think it has to do with the path. When compiling, is there something that I should be doing to make the User Manual file available to the standalone? Right now, the User Manual file is in the same folder as the executable file. In the code for v1, The callback for the User Manual Help menu contains the code: open(‘UserManualv1.pdf’). My other thought is that this needs to be modified, but I do not know what needs to be added/changed.
This is the error I get when using the exe file:
??? Error using ==> open at 127
File ‘UserManualv1.pdf’ not found.
Error in ==> v1>menu_help_userguide_Callback at 2336
Error in ==> gui_mainfcn at 96
Error in ==> v1 at 42
Error in ==>
@(hObject,eventdata)v1(‘menu_help_userguide_Callback’,hObject,eventdata,guidata(hObject))
??? Error while evaluating uimenu Callback
Thanks in advance for any help!
Hello,
I developed a little program that uses the brush tool inside figures. the user of this program should pick some points on a graph and create a variable from it.
After compilation, the brush don’t work properly and when I create the variable out of the points I chose, Matlab is not saving it.
I did not found any listing about problems with the brush function after compilation.
I would be glad if you can help me.
Thanks.
I just learned how to set the path from shortcut.
http://usefulcodes.blogspot.com/2010/01/setting-path-straight-in-matlab.html
This shall allow users to set path to current working directory from single click. Easy for temporary usage on path, I suppose.
Hello Loren,
I’ve got a question that you seem to address, yet I don’t quite see how to sort it out:
My .m function works great from Matlab, but the compiled version reports an error when attempting to access a file-id with textscan. The program makes it past the opening of the file but breaks at the attempting to use the fid within a function.
The code is of this nature:
..continuing..
Also, the function works fine if the filename is explicitly specified in the fopen command rather than passed from the function input.
?
Schaugn,
This error occurs for one of three reasons:
1. The data file has not been included in the deployed application. (Use mcc’s -a switch to add a data file to a deployed application.)
2. The path to the data file is not the same when the application has been deployed. fullfile(pwd,file) only works when the data file exists in the same directory as the executable (this is not typically the case). Instead, just call fopen on the file name: fopen(file). This allows the deployed application to look for the file on the MATLAB path.
3. The application is attempting to write to a non-writable file. (This probably isn’t the case in your application, given the code you’ve posted.)
Try the suggestions in 1 and 2, and let me know if you’re still having trouble.
Hi Peter,
Thanks for the reply..
So, I do not include the datafile with the compilation because the aim is to apply the action on varying folders as they are created. Also, this action will be deployed on a non-matlab machine. The usage is expected to be like:
myprogram('path/filename')So, it seems to me that your recommendation 1) would not be the strategy for this deployment.
I’ve tried a number of ways to pass the ‘path/filename’ argument to the fopen function, yet it only seems to work if I provide a static string manually. I would like myprogram to accept any ‘path/filename’ provided by the user.
I’ve checked pathway interpreted by myprogram and it seems as if the files should be available to open. It seems I’m just not passing it correctly. The exact same .m file works great while the compiled .exe does not.
Works:
> myprogram('path/filename')Does Not Work:
! myprogram('path/filename')Hi Peter,
So.. I read a bit deeper in the link you provided in 9. and managed to sort out the problem..
The problem was actually with the way that I was passing the path/filname string from the system command prompt – I was using the ‘path/filename’ matlab convention rather than the system command prompt convention. Kind of silly.
Hi peter,
I got a question when i use the compiler to a application .exe, it shows “The file ‘.\cvx\matlab6\true.m’ is not in the application’s expanded CTF archive at ‘C:\DOCUME~1\LOCALS~1\Temp\Administrator\mcrCache7.11\SAR_GU10′. This is typically caused by calls to ADDPATH in your startup.m or matlabrc.m files.Please see the compiler documentation and use the ISDEPLOYD function to ensure ADDPATH commands are not executed by deployed application”. I used a function to call a software package. How can solute this problem, thanks!
On the Issue of the Compiler, I am having some difficulty with the change of direction that Mathworks has imposed. In the development of standalone applications using a previous version of the compiler (supplied with 7.0.4a) when the Operator double clicked on app.exe the installer process would extract this and place it into a directory that was relative to the position of the app.exe. In the new compiler suppllied with 7.9.0A when the operator clicks on app.exe the installation appears to be somewhere in the Profiles (usually a tmp/temp) directory.
The Matlab documentation on the compiler does not detail the full extraction process nor the locations.
I would like to know how I can pusuade the compiler / ctf extraction process to place the deployed application where it is intended (i.e. C:Program Files)
On a second issue pertaining to “extractCTF.exe” this application does not function due to ctfrt.dll being reported as missing.
Any help would be appreciated, thanks
Andrew,
If you specify -C when you compile (i.e., mcc -C …) the Compiler will not embed the CTF archive into the application. When the application runs, it will expect to find the CTF archive in directory containing the application, and it will extract the contents into that directory. (This is close to the behavior that you’re looking for, I think, though it means you have to ship two files instead of just one.)
Also, if you’re having trouble with ctfrt.dll, you might check to be sure the directory that contains it (/bin/) is on your PATH.
Finally, please note that you can find the full details of CTF archive embedding and extraction here:
http://www.mathworks.com/help/toolbox/compiler/brl4_f1-1.html
You can set environment variables that control how and where the CTF archive is extracted. You can, for example:
C:\> set MCR_CACHE_ROOT=.
And then run your application — it will extract the CTF archive into the current directory. Use MCR_CACHE_VERBOSE (set it to 1) to see exactly what’s happening during the cache extraction process.
Hi Loren,
I have a GUI consisting of various functions as well as some Sub-folders containing mostly my input & output txt files. I need the organization & can’t just have all the files out of their designated folders. Currently I’m using “cd” command to change the directory, however that’s not working for the compiled version. Is there away to avoid this problem?
Hi Loren,
I am developing a stand-alone application with the GUIDE tool. The application consists of a Graphical User Interface that allows the behavior of different algorithms to be evaluated, not only implemented now, also some that can be further developed. This application must be a stand-alone deployment, so the users can work with this open-source application and evaluates their own algorithms.
The idea is that the user can program their algorithms in a Matlab function template (.m file) (created by myself), and my application calls the template function modified by the user, evaluates the algorithm, shows the results and compares with other algorithms.
So, I used the function feval in order to evaluate the algorithms. The problem is that if I modify the template function, which is the idea of this application, the application does not detect the change in the .m template, and uses the original contents of the .m file.
I use this command: , because is the only way that my application can evaluate the function that contains the algorithms that I created. But, if I want to evaluate another function, developed by another user, it is not possible because it is not referenced in the main code has: .
So, my question is: Is it possible to call a function that has not been reported to the compiler by the command? What other tools exist that help me to develop this application in order to call non-reported functions?
Any help would be appreciated.
Thank you very much.
Hi Loren,
Sorry for the last reply, but the code that I talking about is the following: %#function pragma.
Thank you very much