Usage for MATLAB

Usage for MATLAB#

OptiProfiler provides a benchmark function. This is the main entry point to the package. It benchmarks given solvers on the selected test suite.

We provide below simple examples on how to use OptiProfiler in MATLAB. For more details on the signature of the benchmark function, please refer to the MATLAB API documentation.

Examples#

Example 1: first example to try out#

(See also the file in the repository: matlab/examples/example1.m)

Let us first try to benchmark two callable optimization solvers solver1 and solver2 (e.g., fminsearch and fminunc in MATLAB Optimization Toolbox) on the default test suite. (Note that each solver must accept signatures mentioned in the Cautions part of the benchmark function according to the type of problems you want to solve.)

To do this, run:

scores = benchmark({@solver1, @solver2})

This will benchmark the two solvers under the default test setting, which means 'plain' feature (see Feature) and unconstrained problems from the default problem library whose dimension is smaller or equal to 2. It will also return the scores of the two solvers based on the profiles.

There will be a new folder named out in the current working directory, which contains a subfolder named plain_<timestamp> with all the detailed results.

Structure of the plain_<timestamp> subfolder

Figure 1: Screenshot of the subfolder containing detailed results of the benchmarking run.#

Additionally, a PDF file named summary.pdf is generated, summarizing all the performance profiles and data profiles.

Summary PDF preview

Figure 2: Screenshot of the summary PDF file summarizing all the performance profiles and data profiles.#

The subfolder test_log contains diagnostic files for the experiment. In particular, test_log/report.txt records selected problem names, timing information, and special cases detected while building the profiles: problems where merit_init = phi(x_0) = Inf (all solvers are declared to pass that problem/run), solver runs that terminated abnormally, and solver outputs that were replaced by the initial point as an output-based penalty. The file test_log/log.txt contains the messages printed during the run.

Example 2: one step further by adding options#

(See also the file in the repository: matlab/examples/example2.m)

You can also add options to the benchmark function. For example, if you want to benchmark three solvers solver1, solver2, and solver3 on the test suite with the 'noisy' feature and all the unconstrained and bound-constrained problems with dimension between 6 and 10 from the default problem set, you can run:

options.ptype = 'ub';
options.mindim = 6;
options.maxdim = 10;
options.feature_name = 'noisy';
scores = benchmark({@solver1, @solver2, @solver3}, options)

This will create the corresponding folders out/noisy_<timestamp> and files as in Example 1. More details on the options can be found in the benchmark function documentation.

For the deterministic noisy variant from Moré and Wild’s benchmarking model, set options.noise_mode = 'deterministic'. If options.n_runs is not provided, OptiProfiler uses one run for this deterministic feature unless options.solver_isrand marks at least one solver as randomized, in which case OptiProfiler uses five runs as usual.

options.feature_name = 'noisy';
options.noise_mode = 'deterministic';
options.noise_map = 'chebyshev';
scores = benchmark({@solver1, @solver2, @solver3}, options)

By default, n_jobs is set conservatively to about half of the available workers instead of all workers. For the most reproducible timing experiments, set options.n_jobs explicitly, for example options.n_jobs = 1 for sequential runs.

Example 3: useful option load#

(See also the file in the repository: matlab/examples/example3.m)

OptiProfiler provides a practically useful option named load. This option allows you to load the results from a previous benchmarking run (without solving all the problems again) and use them to draw new profiles with different options. For example, if you have just run Example 2 and OptiProfiler has finished the job and successfully created the folder out in the current working directory, you can run:

options.load = 'latest';
options.solvers_to_load = [1, 3];
options.ptype = 'u';
options.mindim = 7;
options.maxdim = 9;
scores = benchmark(options)

This will directly draw the profiles for the solver1 and solver3 with the 'noisy' feature and all the unconstrained problems with dimension between 7 and 9 selected from the previous run. The results will also be saved under the current directory with a new subfolder named noisy_<timestamp> with the new timestamp.

Example 4: testing parametrized solvers#

(See also the file in the repository: matlab/examples/example4.m)

If you want to benchmark a solver with one variable parameter, you can define function handles by looping over the parameter values. For example, if solver accepts the signature @(fun, x0, para), and you want to benchmark it with the parameter para taking values from 1 to 3, you can run:

solvers = cell(1, 3);
options.solver_names = cell(1, 3);
for i = 1:3
    solvers{i} = @(fun, x0) solver(fun, x0, i);
    options.solver_names{i} = ['solver' num2str(i)];
end
scores = benchmark(solvers, options)

Example 5: customizing the test suite#

(See also the file in the repository: matlab/examples/example5.m)

OptiProfiler allows you to customize the test suite by creating your own feature and loading your own problem library. For example, if you want to create a new feature that adds noise to the objective function and perturbs the initial guess at the same time, you can try the following:

options.feature_name = 'custom';
options.mod_fun = @(x, rand_stream, problem) problem.fun(x) + 1e-3 * rand_stream.randn(1);
options.mod_x0 = @(rand_stream, problem) problem.x0 + 1e-3 * rand_stream.randn(problem.n, 1);
scores = benchmark({@solver1, @solver2}, options)

If you want to benchmark solvers based on your own problem library, you should do the following three steps:

  1. Create a new subfolder (e.g., 'myproblems') within the 'problems' folder located in the optiprofiler project root directory.

  2. Implement two MATLAB functions:

  • <your_problem_library_name>_load.m: Define a function that accepts a character or string representing the optimization problem name and returns a Problem class object (see Problem).

  • <your_problem_library_name>_select.m: Define a function that accepts a structure to specify desired problem characteristics and returns a cell array containing names of all problems in your library that satisfy the requirements. The structure may include fields such as ptype, mindim, maxdim, minb, maxb, minlcon, maxlcon, minnlcon, maxnlcon, mincon, maxcon, and excludelist (these fields descriptions can be found in the benchmark function documentation in “Options for problems” part).

  1. Use the benchmark function as before, but specify your desired problem libraries. For example, to use both the default S2MPJ library and your custom library in the subfolder 'myproblems', you can run:

options.plibs = {'s2mpj', 'myproblems'};
scores = benchmark({@solver1, @solver2}, options)

You may also refer to the README file in the 'problems' folder for a detailed guide on how to create and use your own problem library via the OptiProfiler package.

Example 6: wrapping solvers with nonlinear constraints#

(See also the file in the repository: matlab/examples/example6.m)

For nonlinearly constrained problems, OptiProfiler calls each solver with the signature

x = solver(fun, x0, xl, xu, aub, bub, aeq, beq, cub, ceq)

where cub(x) <= 0 contains the nonlinear inequality constraints and ceq(x) = 0 contains the nonlinear equality constraints. MATLAB solvers such as fmincon instead expect one nonlinear constraint callback nonlcon returning both values. The small but important adapter is deal: the expression @(x) deal(cub(x), ceq(x)) evaluates OptiProfiler’s two callbacks and returns them as the two outputs expected by fmincon, i.e., [c, ceq] = nonlcon(x). See the MathWorks documentation for fmincon nonlinear constraints and deal.

function x = fmincon_short(fun, x0, xl, xu, aub, bub, aeq, beq, cub, ceq)
    x = fmincon_wrapper(fun, x0, xl, xu, aub, bub, aeq, beq, cub, ceq, 100);
end

function x = fmincon_long(fun, x0, xl, xu, aub, bub, aeq, beq, cub, ceq)
    x = fmincon_wrapper(fun, x0, xl, xu, aub, bub, aeq, beq, cub, ceq, 200);
end

function x = fmincon_wrapper(fun, x0, xl, xu, aub, bub, aeq, beq, cub, ceq, max_fun_evals)
    % Convert OptiProfiler's separate nonlinear callbacks to fmincon's
    % two-output callback: [c, ceq] = nonlcon(x).
    nonlcon = @(x) deal(cub(x), ceq(x));
    options = optimoptions('fmincon', 'MaxFunctionEvaluations', max_fun_evals);
    x = fmincon(fun, x0, aub, bub, aeq, beq, xl, xu, nonlcon, options);
end

Then pass the wrappers to benchmark as ordinary solvers:

options.ptype = 'n';
options.problem_names = {'HS10', 'HS11', 'HS12'};
options.plibs = {'s2mpj'};
options.mindim = 2;
options.maxdim = 5;
options.max_eval_factor = 500;
options.draw_hist_plots = 'none';
options.n_jobs = 1;
options.solver_names = {'fmincon short', 'fmincon long'};
scores = benchmark({@fmincon_short, @fmincon_long}, options)