This week, guest blogger Peter Webb continues his series of articles about the MATLAB Compiler. This week's topic: managing MATLAB's paths in a compiled application. For an introduction to writing deployable code, please see the June 19th article.
Contents
One of the goals of the compilation process is to turn a flexible and easily modifiable MATLAB progam into a robust software component with consistiently predictable behavior: a compiled application should not be able to change which functions it calls or the way those functions work. As a result, certain aspects of the execution environment that are malleable in MATLAB become fixed or constant in a deployed application. Perhaps the most important of these is the mechanism by which MATLAB locates functions and data files: MATLAB's search paths.
A path consists of a list of directories that MATLAB searches to find files. MATLAB typically searches a path in list order, stopping at the first file that matches the search criteria. MATLAB uses paths for two reasons: to determine which functions to execute; and to locate data files in the file system.
MATLAB primarily interacts with four paths, two internal and two external: the MATLAB path; the MATLAB Java class path; the system path; and the system load library path.
Most familar, perhaps, is the MATLAB path, which affects how MATLAB determines which MATLAB function files to run and which MAT-files to open. Though the MATLAB path is the most visible, of nearly equal importance is the MATLAB Java class path, which specifies the directories in which MATLAB searches for Java functions. The two external paths list the locations which the operating system will look for executables and shared libraries, respectively. The Windows operating system uses the same path to look for executables and shared libraries, but most Unix systems search separate paths for these two types of files.
When writing an application intended for deployment via the MATLAB Compiler, keep in mind that MATLAB applications typically interact with paths in at least three independent ways:
- Executing MATLAB commands to change a path
- Passing paths as arguments to MATLAB functions
- Relying on the existence or structure of external paths
After briefly listing some general guidelines for safely interacting with paths in MATLAB, I provide more detailed techniques to manage problems that can occur in each of these areas.
General Guidelines
A compiled application behaves more consistently if it:
- Does not change any path during execution.
- Uses relative paths (or anchors paths to a known root via the matlabroot or ctfroot functions).
- Avoids accessing or changing the current directory.
These guidelines become more important when the compiled application executes in an envrionment that differs from the one in which it was developed: for example, when installed at a customer site.
In cases where it is convenient to keep path management commands in your M-files when they run in MATLAB, you can use the isdeployed function to skip over calls to path management functions in deployed applications. For example, this M-code adds my "beta"-quality M-files to the path when my applications run in MATLAB:
if ~isdeployed addpath /home/pwebb/mfiles/beta end
MATLAB Commands that Directly Access the Path
All M-file and Java functions rely on the state of MATLAB's internal paths, but only a few MATLAB commands directly access or modify these paths. Most of these functions should be avoided in deployed applications because they have the potential to change the way the application works, either by changing the functions it calls or the data files it loads.
| addpath: | Add a directory to the MATLAB search path. |
| cd: | Change the current directory. |
| javaaddpath: | Add a directory to MATLAB's Java search path. |
| javaclasspath: | Get or set MATLAB's Java search path. |
| javarmpath: | Remove a directory from MATLAB's Java search path. |
| path: | Get or set the MATLAB search path. |
| rmpath: | Remove a directory from MATLAB's search path. |
| savepath: | Save the current MATLAB path. |
Path management commands often appear in startup or initialization code; when it comes time to deploy your application, you may have forgotten that the application relies on these path settings. Problematic path settings sometimes occur in these oft-forgotten files:
| pathdef: | Used by MATLAB to establish the initial MATLAB search path, pathdef.m may be modified, in particular by the savepath command. |
| startup: | The MATLAB Compiler incorporates startup.m into every generated executable and shared library; any path management commands in startup.m therefore need to work in the deployed application. Directories added to MATLAB path by startup.m are automatically added to the MATLAB path in a deployed application by the MATLAB Compiler if the deployed application uses any files from those directories, thus making the addpath calls in startup.m both redundant and dangerous. |
Protect all calls to path management functions in these files by enclosing them in an ~isdeployed if-block.
Instead: Ask yourself why you're changing the path. Rename functions or use MATLAB Objects to manage your function namespace. Change the path before running the MATLAB Compiler to add optional or conditional functionality to your application.
Instead: Use paths relative to matlabroot or ctfroot.
Change code like this:
cd my/data/directory fp = fopen('data.file', 'r');
To code like this:
fp = fopen(fullfile(ctfroot, 'my', 'data', 'directory', 'data.file'));
Instead: Set the Java class path before running the MATLAB Compiler. Use classpath.txt or javaaddpath.
Instead: If you must save the path, save it to a MAT-file; otherwise, enclose the call to savepath in an ~isdeployed if-block.
p = path; save savedpath p
As further incentive to avoid these functions in deployed applications, please note the commands that change the Java class path (javaaddpath, javarmpath) also clear the values of global variables.
For more details on how to manage the MATLAB function namespace with MATLAB objects, please see the polynomial example in the MATLAB documentation.
Paths as Arguments to MATLAB Functions
Many MATLAB programs assume that the directory structure of the filesystem cannot change. When this assumption leads to the use of absolute paths in file names, e.g., load('c:/Work/MATLAB/data.mat') or an implicit dependency on MATLAB's ability to locate a file, e.g., load data.mat, it creates problems for deployed applications. A deployed application is very likely going to be installed on a machine that has neither the same directory structure nor the access to the same filesystem as the machine on which it was developed. The use of absolute paths causes errors because of differences between the development and the deployment file system structures. But relying entirely on relative paths establishes a fragile dependency on the location of the current directory. The solution: base all file locations on a known root.
In MATLAB, there are typically two file system roots that matter: the root of the external file system, typically / or c:/ on Unix and Windows systems respectively; and the MATLAB root, the directory in which MATLAB has been installed. In a deployed application there is an additional root: the CTF root, which is the location of the application's MATLAB content (M-files, data files, JAR files -- everything that the MATLAB Compiler put into the application's CTF file). In MATLAB ctfroot and matlabroot refer to the same directory: MATLAB's installation directory. In a deployed application, matlabroot refers to the directory in which the MATLAB Common Runtime (the MCR) is installed and ctfroot to the CTF root.
Deployed applications typically base their file names on the the CTF root. For example, assume an application that reads and writes MAT files. To ensure that the MATLAB Compiler includes the input data files in the CTF archive, specify the data files with mcc's -a (add) flag:
mcc fcn.m -a data.mat
In the application code, use which or ctfroot to construct a full path to the files being loaded or saved.
Use which when a known filename exists on the application's path:
[pathstr,name]=fileparts(which('data.mat')); % Load x from the MAT file load(name); x = x + 17; % Save x back to the MAT file save(fullfile(pathstr, 'data.mat'), x);
Use ctfroot when creating a new file:
data = input('Enter data: '); save(fullfile(ctfroot, 'more_data.mat'), data, '-append');
This use of ctfroot assumes that the application has permission to write to the file system on the deployment machine. This is not always true by default, especially with network installations. Work with your IT department to secure the appropriate file system access.
Relying on the Structure of External Paths
A MATLAB application that uses the system or loadlibrary functions or calls MEX files linked to non-MathWorks shared libraries has a dependency on the external system paths. The system command allows MATLAB programs to directly execute operating system commands and capture their output as text. This can be very convenient, but it of course requires that the command be available in the current file system. loadlibrary allows MATLAB programs to load shared libraries and call their exported APIs; it similarly requires that the library in question be available on the deployment machine's file system.
The techniques for managing external path dependencies are very similar to those used to manage internal path dependencies: use full paths, favor explicit dependencies over implicit ones, and include all required files in the deployed application.
- Use full path names, rooted in the ctfroot, to specify the arguments to the system and loadlibrary functions.
- Include non-standard applications and shared libraries in the deployed application by using mcc's -a switch.
- Ensure the necessary applications and libraries are present on the deployment machine or in a network location accessible from the deployment machine.
Next Up: Non-supported functions
The next post in this series will explore mechanisms for identifying and managing calls to non-supported functions. In the meantime, you can refer to the documentation for the MATLAB Compiler or post follow-up questions here.
Get
the MATLAB code
Published with MATLAB® 7.6
Hi Loren and Peter,
I have a software package that I currently deploy as a standalone application. I am currently considering releasing some of the matlab code as open source, so that matlab-having clients can develop plugins for my package, but I would like to protect some of the code, for which I can use p-code, but as importantly, I would like to make sure that nobody can compile a standalone application from the open source code I release. The open source permissions for these clients would only extend to within matlab.
The first, obvious solution is to create a critical p-file that checks isdeployed(), and won’t run if it is deployed. The problem I see with this is that isdeployed() can be overshadowed by the user, prior to compilation. Is there a solution to this vulnerability? How can I release truly undeployable .m/.p code? How does The Mathworks protect imtool, etc?
Thanks for a great column, and any help would be greatly appreciated!
Great post Peter!!
The fullfile tip is very useful, I’m sure lots of people will benefit from this post. As a newbie MATLAB programmer, I fell victim to many of the scenarios you listed here. Those changes in path can really be troublesome!
Quan
Compiling data with a deployed application is generally not a good idea. A better approach would be to make the data an input to the application. The path to the data would be supplied by the user of the application either from the command line or using a gui.
Johan,
The MATLAB Compiler doesn’t currently provide any user-accessible ways of preventing the deployment of M-files. We’ve been thinking about this though; it’s a tough problem, which is why you haven’t seen a solution from us yet.
For now, you’ll have to manage this issue yourself. You’ll need to decide exactly how much security you want; the more you want, the more work you’ll need to do. Some ideas you might try:
* Use isdeployed() as you outlined above; yes, it can be overloaded, but how many people will think to do that?
* Use a license manager, like FlexLM, for example, and develop a MEX file to interact with it. Wikipedia has some ideas here: http://en.wikipedia.org/wiki/License_Management.
* Rely entirely on your license agreement with your users, and do not attempt a technical solution.
Keep in mind that there are probably effective attacks on all of these suggestions — there’s no such thing as a perfectly secure system.
Fundamentally, you’re up against a very difficult problem. It’s very hard to simultaneously distribute source code and control how that source code can be used.
Regarding overloading of isdeployed(), can’t you use builtin(’isdeployed’) as a workaround? (Although I guess builtin could also be overshadowed…)
Hi Peter,
Thanks for bringing up the idea of using ctfroot to make the paths deterministic. However, not only file permissions is a big problem for my project, this also makes it difficult to manage deployed machines when the user have to look under something like
C:\Documents and Settings\Username\Local Settings\Application Data\MathWorks\mcr_cache_v79\ICDviewer_63A7BF5EA103F85B3934BF5A0DE7D64D\data
to find a .mat to send it to the developer for problem tracking!
Is there a way that the MCR can extract the guts (encrypted .m files) according to the same folder structure when it was compiled (i.e. where the .exe is located) and extract the rest (toolboxes and all supporting files) /mcr_cache right under it?
If I were to base my paths off a pivot, it should be the folder where the compiled executable is located on, not some random place on the drive.
If there’s a good rationale for the path management scheme to be like the way it is right now that will benefit the users, please let us know, and I’m sure we can change our habits. On the other hand, if designing it the other way round is difficult, please discuss the difficulties and we’ll mark (comment) our code and wait for the future version.
I also think that a function which returned the path of the deployed exe file would be very useful. As Hoi mentioned above, the return of “ctfroot” is some rather strange path that I find very little use for. I usually solve the problem by creating an installer that sets a system variable to the root directory of the standalone application.
Gunnar: Actually you don’t have to rely on changing the MCR path (although it’s a good idea because the default paths is too long, long enough to break some old DOS program like deltree.exe) and complicate the installation/maintanence process. You can just use setappdata(0, ‘rootFolder’, pwd) at your main program and retrieve the path pivot anywhere.
However, I still don’t like the idea of having MCR messing with the my path structures. This only adds unnecessary mental gymnastics to the programming task, hurting productivity.
If a programming language feature is clearly unintuitive to users like the MCR paths, it better has a good justification other ‘it’s easier to develop the compiler that way’.
Hoi and Gunnar,
You can exercise more control over where the encrypted files are installed via the Compiler’s -C switch or the MCR’s “component cache” environment variables.
The details: http://www.mathworks.com/access/helpdesk/help/toolbox/compiler/brl4_f1-1.html
If you place the CTF archive and the executable in the same directory at installation, then CTFROOT will return the directory containing the executable.
With -C switch, I think ctfroot will be the cleanest solution as the root path. Thanks Peter!
Even with the -C switch, the MCR still breaks my path structure as now /myProgram/core is extracted to /myProgram/myProgram_mcr/core, but it’s better than having it extracted to somewhere deep under ‘local settings’.
To shorten development time, I took a chance on MATLAB compiler when I decided to use MATLAB for a software project meant for distribution, and I found a bunch of things that ought to be right (DCOM, internationalization, single instances, uitable, paths, etc.) that wasn’t. I filed bug/enhancement requests for all of them, but it will take a long while before the developers actually do something about it.
I really hope Mathworks can invest more on developing MATLAB compiler to make it a more polished product. With cheap dual-core computers @1GB RAM minimum ready to run MCR, developing applications (especially GUI with data visualization features) using MATLAB is a competitive alternative to C#/Java.
Hi,
We have some need about a deployement process. We have a application which is compiled by MatLab Compiler last release, V4.8.
When we are in development or in deployement, we need to know the root directory from where the application was launch (localisation of the exe directory).
It’s like CTFROOT but for the EXE.
Do you know if it is possible to have that ?
Thanks a lot.
Best regards.
GALLOU,
I think the information in comment #9 might help you as well. If you compile using -C, and place the CTF archive in the same directory as the executable, then CTFROOT will return the directory containing the executable.
Hoi,
Long-term, we certainly plan on further improving the quality of the MATLAB Compiler (and all of the Builder tools). I’m not sure exactly which bugs you’re referring to; I took a look in our list and didn’t find any mention of uitable, but I noticed several internationalization and COM bug fixes going into the next release, so perhaps we’ve already addressed some of your concerns.
Hi Peter,
I’m glad to hear that you guys have plans to improve the compiler quality. Quality-wise, I think most people can live with minors bugs if there’s a workaround, but if the scheme/idea is not well-thought out (like single instances and default settings like Windows Vista that drives user nuts), it might start to shake people’s confidence about whether they’re using the right tools.
I was tempted to scratch my project and switch to C# for 4~5 times when I had to fight with the MATLAB compiler, and thanks to the quick and helpful support, I got through most of them and was back on track. The most fustrating experience was DCOM: the apps engineer sent me a documentation page that remote DCOM is not supported by the compiler, and I ended up setting up two virtual machines, taking the difference in the Windows+/-MATLAB/MCR installations and found that it’s just one single line that I have to add to get MCR to use remote DCOM! If that’s the one that you’re talking about, I look forward to remove my ’secret sauce’ in the deployment installer.
For internationalization, if all the developer’s machine are set to display ‘Japanese’ for non-Unicode programs, the daily problems in compiled programs will annoy them enough for them to fix it (even if the developer can read Japanese, the characters are still messed up for half of the times). If the developers assume users set the non-Unicode programs to display Japanese because they can read Japanese (bilingual), then I am hoping that at least if MCR chose to display Japanese error messages in the console, display it properly in Japanese code page. Of course, it’d be the best if English MATLAB consistently shows everything in English, in any OS.
For uitable, an apps engineer called (a long time ago) and promised to work on it, but I hadn’t heard back from anybody ever since. The request ID is 1-5Y36TS. Basically the uitable() in my compiled program displays blank until I refresh it again with something else, and that doesn’t happen to the full MATLAB. We can take this discussion offline for better nettiquette.
I feel MATLAB compiler can expand its territory a lot more. If Steve can have a blog on something as specific as Image processing, I certainly appreciate a blog on Application Deployment branch of MATLAB products, as they can be quite obscure and daunting and maybe a community can help promoting the product through more success cases.
Hi,
I have created a standalone executable (for Windows), called ‘v1′, that has a Help Menu. I want the user to be able to use the Help Menu to open a pdf User Manual, but get an error when trying to do this from the deployed/compiled version. I think it has to do with the path. When compiling, is there something that I should be doing to make the User Manual file available to the standalone? Right now, the User Manual file is in the same folder as the executable file. In the code for v1, The callback for the User Manual Help menu contains the code: open(’UserManualv1.pdf’). My other thought is that this needs to be modified, but I do not know what needs to be added/changed.
This is the error I get when using the exe file:
??? Error using ==> open at 127
File ‘UserManualv1.pdf’ not found.
Error in ==> v1>menu_help_userguide_Callback at 2336
Error in ==> gui_mainfcn at 96
Error in ==> v1 at 42
Error in ==>
@(hObject,eventdata)v1(’menu_help_userguide_Callback’,hObject,eventdata,guidata(hObject))
??? Error while evaluating uimenu Callback
Thanks in advance for any help!
Hello,
I developed a little program that uses the brush tool inside figures. the user of this program should pick some points on a graph and create a variable from it.
After compilation, the brush don’t work properly and when I create the variable out of the points I chose, Matlab is not saving it.
I did not found any listing about problems with the brush function after compilation.
I would be glad if you can help me.
Thanks.