Usage for MATLAB#
OptiProfiler provides a benchmark function. This is the main entry point to the package. It benchmarks given solvers on the selected test suite.
We provide below simple examples on how to use OptiProfiler in MATLAB. For more details on the signature of the benchmark function, please refer to the MATLAB API documentation.
Examples#
Example 1: first example to try out#
(See also the file in the repository: matlab/examples/example1.m)
Let us first try to benchmark two callable optimization solvers solver1 and solver2 (e.g., fminsearch and fminunc in MATLAB Optimization Toolbox) on the default test suite. (Note that each solver must accept signatures mentioned in the Cautions part of the benchmark function according to the type of problems you want to solve.)
To do this, run:
scores = benchmark({@solver1, @solver2})
This will benchmark the two solvers under the default test setting, which means 'plain' feature (see Feature) and unconstrained problems from the default problem library whose dimension is smaller or equal to 2. It will also return the scores of the two solvers based on the profiles.
There will be a new folder named out in the current working directory, which contains a subfolder named plain_<timestamp> with all the detailed results.
Figure 1: Screenshot of the subfolder containing detailed results of the benchmarking run.#
Additionally, a PDF file named summary.pdf is generated, summarizing all the performance profiles and data profiles.
Figure 2: Screenshot of the summary PDF file summarizing all the performance profiles and data profiles.#
The subfolder test_log contains diagnostic files for the experiment. In
particular, test_log/report.txt records selected problem names, timing
information, and special cases detected while building the profiles: problems
where merit_init = phi(x_0) = Inf (valid evaluations pass by convention,
but undefined objective or constraint values do not), solver runs that
terminated abnormally, and solver outputs that
were replaced by the initial point as an output-based penalty. The file
test_log/log.txt contains the messages printed during the run.
If history plots are enabled, each problem library has a subfolder under
history_plots. The top-level PROBLEM.pdf files keep the combined view
with both raw and cumulative-minimum histories. The raw and cummin
subfolders contain the corresponding single-view PDFs. The merged
*_history_plots_summary.pdf file uses the top-level combined PDFs.
History plots with extreme values#
History plots use a display copy of the recorded values. In both Python
and MATLAB, finite values outside [-1e100, 1e100] are clipped to that
interval before calculating the displayed means and error bands. This
large plotting limit leaves room for squared deviations, translations and
axis margins without floating-point overflow. It is not an assumption that
all valid objective values lie in that interval.
Clipping diagnostics remain available in the experiment report and structured plot data, but are not drawn inside history panels. NaN and Inf entries are counted separately and displayed above the finite range of the corresponding run (including its finite initial value). If that run has no finite history or initial value, their display placeholder is 1. A constant finite range gets a small positive gap so its nonfinite placeholders remain distinguishable. These placeholders are not successful function evaluations.
The clipping occurs before both raw-view and cumulative-minimum-view statistics. The term “raw” in a plot filename denotes the non-cumulative view, not an unmodified serialization of the experiment. Original saved histories, oracle evaluations, output points, merit values and profile scores are unchanged. For numerical analysis of extreme values, use the saved data rather than recovering values from a capped plot.
Positive tiny values are not replaced by a fixed display floor. Nonpositive
histories are shifted when a logarithmic view is needed, and the shift is
shown in the axis label. Logarithmic limits and extreme-range ticks are kept
within representable floating-point values, so a finite history cannot be
silently replaced by a default [1, 10] viewing range.
For errorbar_type='meanstd', the two interfaces retain their established
standard-deviation conventions across the N runs: Python divides the
sum of squared deviations by N (population normalization), while MATLAB
divides by N-1 when N > 1 (sample normalization). With one run,
both use zero standard deviation. The same convention is used to select the
logarithmic shift and to draw the band within each interface. Therefore,
unequal-run meanstd bands, and sometimes their displayed shifts, are
not expected to match exactly between Python and MATLAB. This compatibility
choice does not change either interface’s saved values or profile scores.
MATLAB output robustness#
Normal MATLAB sessions use native figures and PDF output. The -batch and
-nodesktop flags do not disable the Java Virtual Machine (JVM).
On macOS and Linux, OptiProfiler also supports saving without a JVM:
MATLAB releases with working native graphics (verified with R2026a) retain the native PDF route.
If MATLAB cannot create a figure, as in R2023b with
-nojvm, OptiProfiler writes plain, browser-readable SVG charts andsummary.htmlinstead of PDF/FIG. These use the same computed profile curves and history preprocessing; they do not promise native styling, LaTeX interpretation, or interactive FIG files. Mean and errorband values remain the same; the portable history chart labels errorband boundaries as separate lines.A failing native PDF exporter can also use SVG as its last fallback. If only PDF merging fails, existing individual plots remain available through an HTML index and the complete backend diagnostics are saved.
The initial native-graphics probe is cached for the MATLAB process, avoiding a new probe figure for every plot. A transient failure of that initial probe therefore keeps subsequent plots on the SVG route until MATLAB is restarted.
On Linux with MATLAB R2026a, a separate safety check avoids a reproducible
native-export hang in the embedded Chromium renderer. Its observed temporary
socket name adds 46 bytes to TMPDIR, while a Linux Unix-domain socket path
allows at most 107 bytes. An explicit temporary path longer than 61 UTF-8
bytes (ignoring trailing separators) therefore uses the existing SVG/HTML
route instead of native PDF/FIG.
Relative TMPDIR values also use this conservative fallback; use a short
absolute path for native output. This does not claim every relative path
fails in MATLAB. An unset TMPDIR uses the normal system default. An explicitly
empty TMPDIR also reproduced the hang, so it uses the portable route instead.
When MATLAB reports an empty value, a small read-only shell check distinguishes
unset from set-empty without requiring Java; an unsuccessful check also falls
back with an explicit diagnostic.
The message explains the fallback. To request native output, set a short
absolute temporary path, for example setenv('TMPDIR', '/tmp'), before
calling benchmark. OptiProfiler does not change that environment variable
or move experiment outputs itself. The path check is repeated on each graphics
capability query, so correcting the temporary path does not permanently retain
the unsafe-path fallback. The guard is specific to the tested Linux R2026a
backend; it is not a general timeout for arbitrary native graphics failures.
It also cannot help when an even longer temporary path prevents MATLAB itself
from starting; shorten that path before launching MATLAB.
When upgrading MATLAB, recheck the native exporter at the 61/62-byte boundary
and with unset versus set-empty TMPDIR before extending this release guard.
The SVG axes explicitly label any logarithmic coordinate transformation.
The display-only history limit and nonfinite replacements are documented in
the reports, without in-panel notes; raw saved values and scores are never
clipped for rendering.
score_only=true neither allocates figures nor writes fallback artifacts,
even if summary options are enabled.
No-JVM file safety is implemented with exclusive POSIX directory creation and
a same-directory temporary file followed by atomic rename on macOS/Linux.
The JVM path retains Java’s atomic file operations. Windows no-JVM output is
not supported: OptiProfiler reports an error asking you to restart MATLAB
without -nojvm, rather than using an unsafe replacement operation. Normal
MATLAB startup enables the JVM; use usejava('jvm') to check your session.
The restriction applies to saving output, not score_only=true evaluation.
External PDF mergers (qpdf, pdfunite, or Ghostscript)
are optional fallbacks, not silently installed dependencies.
The full MATLAB regression suite additionally requires pdfunite and
pdftotext (Ubuntu package poppler-utils) to test actual PDF merging,
page order, and preservation of an existing summary when a merge fails.
Its CI job installs and checks these tools explicitly rather than relying
on MATLAB’s bundled Java classes. This is a test prerequisite, not a new
requirement for users: the no-merger HTML fallback is tested separately.
On Linux CI, only these two system-tool subprocesses clear
LD_LIBRARY_PATH so they do not load an incompatible MATLAB-bundled C++
library. MATLAB’s own environment and user-installed tool configurations
are not changed by OptiProfiler.
In a normal Linux session, PDF merging first tries each tool with the caller’s
environment unchanged. If a tool fails with a specific libstdc++.so
GLIBCXX or CXXABI required-version diagnostic, OptiProfiler retries
that tool once with LD_LIBRARY_PATH removed only in its child process.
The first successful isolated retry prints an informational message. Missing
tools, invalid PDFs, and other errors do not trigger this retry. If no merger
succeeds, the warning and saved diagnostics explain the failures; individual
plots and the HTML index remain available, and a previous summary PDF is not
replaced with a failed attempt. No tool is installed automatically, and no
experiment directory, raw data format, or PDF naming convention is changed.
The full-unit CI entry writes JUnit results, runner diagnostics, and coverage outside disposable test fixtures. CI uploads these receipts even when a test fails; an empty upload is an error, not evidence of a successful test run.
Completed experiment data is saved as a verified version-7.3 MAT file before sequential history rendering. A timestamp becomes loadable only after this save succeeds. Requested saving failures are errors, including in silent mode; completed staging files are retained if final publication fails. This is final-result preservation, not a checkpoint/resume mechanism for an unfinished solver run or a guarantee against abrupt machine/disk failure.
Library-selection failures stop with their original diagnostic instead of
being treated as an empty or unfiltered selection. OptiProfiler restores the
caller’s warning settings on success and exceptions, and borrows an existing
parallel pool without deleting or resizing it. A smaller existing pool may
use fewer workers than n_jobs; this changes throughput, not the requested
problems or evaluation budgets.
If setup cannot save the default pathdef and no default startup location exists, it keeps the installed paths available in the current session and prints the commands needed for future sessions. It does not claim successful persistence. This session-only fallback is not used when a persistence target was explicitly specified. Uninstall removes only paths recorded as newly added by setup, including the bundled S2MPJ runtime paths; existing paths are borrowed and preserved.
Example 2: one step further by adding options#
(See also the file in the repository: matlab/examples/example2.m)
You can also add options to the benchmark function. For example, if you want to benchmark three solvers solver1, solver2, and solver3 on the test suite with the 'noisy' feature and all the unconstrained and bound-constrained problems with dimension between 6 and 10 from the default problem set, you can run:
options.ptype = 'ub';
options.mindim = 6;
options.maxdim = 10;
options.feature_name = 'noisy';
scores = benchmark({@solver1, @solver2, @solver3}, options)
This will create the corresponding folders out/noisy_<timestamp> and files as in Example 1. More details on the options can be found in the benchmark function documentation.
For the deterministic noisy variant from Moré and Wild’s benchmarking model,
set options.noise_mode = 'deterministic'. If options.n_runs is not
provided, OptiProfiler uses one run for this deterministic feature unless
options.solver_isrand marks at least one solver as randomized, in which
case OptiProfiler uses five runs as usual.
options.feature_name = 'noisy';
options.noise_mode = 'deterministic';
options.noise_map = 'chebyshev';
scores = benchmark({@solver1, @solver2, @solver3}, options)
By default, n_jobs is set conservatively to about half of the available
workers instead of all workers. For the most reproducible timing experiments,
set options.n_jobs explicitly, for example options.n_jobs = 1 for
sequential runs.
Example 3: useful option load#
(See also the file in the repository: matlab/examples/example3.m)
OptiProfiler provides a practically useful option named load. This option allows you to load the results from a previous benchmarking run (without solving all the problems again) and use them to draw new profiles with different options. For example, if you have just run Example 2 and OptiProfiler has finished the job and successfully created the folder out in the current working directory, you can run:
options.load = 'latest';
options.solvers_to_load = [1, 3];
options.ptype = 'u';
options.mindim = 7;
options.maxdim = 9;
scores = benchmark(options)
This will directly draw the profiles for the solver1 and solver3 with the 'noisy' feature and all the unconstrained problems with dimension between 7 and 9 selected from the previous run. The results will also be saved under the current directory with a new subfolder named noisy_<timestamp> with the new timestamp.
Reusable features, replay, and retained results#
A Feature is a reusable specification, not an experiment or a random-stream
checkpoint. Its effective stages contain only feature-local options. For example:
stages = { ...
struct('name', 'noisy', 'options', struct('noise_level', 1e-3)), ...
struct('name', 'perturbed_x0', 'options', struct('perturbation_level', 1e-3))};
options = struct('feature', Feature(stages), 'n_runs', 3, 'n_jobs', 1);
scores = benchmark({@solver1, @solver2}, options);
Pass either feature or feature_name, not both. n_runs belongs to the
benchmark options, not to a stage. Reusing the same Feature in another benchmark
does not reuse its previous runtime state. Identity stages such as plain are
normalized away. Repeated nonidentity stages retain their order and separate
options. With run_plain=true, the primary experiment uses its own run count;
the plain-reference role retains the independent one-run policy. A saved run
axis is not a count of actual solver calls: deterministic runs can have copied
slots, which reports identify separately from actual executions.
A Problem may carry an optional feasible reference fact
in its reference property: one finite scalar merit with its kind
('lower_bound', 'optimum', 'best_known' or 'target', each a
claim over the feasible points of the problem), its source and the
mapping token 'feasible_objective/1' of a closed registry. It is author
or provider metadata and is unrelated to the plain-reference role above.
The reference of a FeaturedProblem is that
record, unchanged, if the feature retains it, and [] (unknown) otherwise;
the record holds no point, so nothing is transported or derived, and the rule
is the same for every kind. Stages that change only observations (noisy,
truncated, random_nan, nonquantifiable_constraints,
unrelaxable_constraints, and quantized with ground_truth=false),
perturbed_x0, permuted and linearly_transformed retain it.
custom retains it only if its options are a subset of mod_x0 and
mod_affine; mod_fun, mod_cub, mod_ceq, mod_bounds,
mod_linear_ub and mod_linear_eq may change values, constraints or
bounds and make it unknown, whatever the callbacks do. quantized with
ground_truth=true, the default, makes it unknown, because the truth is then
the mesh problem. A composition retains the record only if every stage does.
The reference fact is not a run-history minimum and not the profile baseline:
that baseline is the least merit observed over the selected solver histories,
changes with the solver cohort and is never stored in a problem; nothing
derives a reference from solver output, and no profile formula reads the
reference. It is not a floor for run merits either. On a constrained problem
the merit of a run may be below the reference, because a merit function
tolerates or penalizes small violations, so an infeasible point can have a
lower merit than every feasible one; such values are legitimate and are never
clamped. The scalar is an objective value over feasible points: before a
consumer compares run merits with it under a custom merit_fun, that
function must be known to satisfy merit_fun(f, 0, maxcv_init) == f for
every maxcv_init (the default merit function does).
There are two different ways to use an existing experiment:
benchmarkwithloadreanalyses saved numerical histories without executing solvers or loading the original provider problems. Current filters describe this selection; they are not a recovered original experiment plan. MATLAB keeps its existing sorted-uniquesolvers_to_loadselection rule.loadBenchmarkOptionsimports trusted native settings for a new benchmark. The caller supplies its solvers and output destination explicitly:[options, receipt] = loadBenchmarkOptions('test_log/options_refined.mat'); options.benchmark_id = 'fresh-replay'; options.savepath = pwd; scores = benchmark({@solver1, @solver2}, options);
New options_refined-v2 files retain the canonical native Feature and a
separate n_runs. The effective feature_specification is inspection-only
when the native Feature is present; inconsistent ordinary values, structure,
or callback positions/types are rejected. Native anonymous functions and
closures need not compare equal after a MAT round trip, so callback identity
and semantics are not compared. The returned receipt explicitly records this
limit, and only the canonical Feature supplies executable callbacks.
Canonical native state preserves resolved options and a separately retained
declaration, including an unknown declaration after a supported legacy import.
It does not reconstruct an original invocation route or resume a live runtime.
Saved output paths, load selectors and envelope fields are not forwarded by the
replay helper. Specification-only v2 structs are accepted as explicit new replay
inputs, not evidence of the original declaration.
Options saved by a load (re-plot) invocation describe that re-plot, not the
archived execution: benchmark writes no options_refined.mat for a load,
and loadBenchmarkOptions rejects an unversioned source with a nonempty
load field (the options_user.mat of a load, or a flat
options_refined.mat that older versions wrote for a load) with
OptiProfiler:LoadInvocationNotReplayable instead of turning the load label
into a replay. A versioned options_refined-v2 record carries its own
effective feature identity, so it is accepted as an explicit replay input and
its obsolete load/solvers_to_load selectors are simply not forwarded.
Replay the source experiment from its own test_log/options_refined.mat.
Old flat native options sometimes omitted feature identity. Such a file requires
an explicit second argument, for example loadBenchmarkOptions(path, 'noisy');
neither folder names nor stamps are used to guess it. The receipt records this
as a present replay override, not a recovered historical request. Native MAT
files are trusted inputs and require their original callback/class dependencies;
they can execute code during loading. JSON callback descriptions cannot be used
to reconstruct executable functions.
New evaluation reports use optiprofiler.eval_report/2 and archive provenance
uses feature_pipeline-v3. Effective feature settings and per-role experiment
plans are separate. Historical payloads retain their original version and
unknown facts; a load report does not turn today’s defaults into old execution
facts. JSON stage positions and same-kind occurrence numbers are zero-based;
MATLAB numerical solver/run indexes remain one-based as recorded by the report.
Complete observed runtime receipts stay in native archives, while compact
reports omit repeated per-stage runtime details. Long or private feature text
may be omitted from JSON with explicit UTF-8 byte counts and reasons; complete
native stamps remain available in native archives/settings.
Report feature declarations and effective stage lists longer than 256 entries
use a bounded object with values (the first 256 entries), total_items,
and reason='metadata_item_limit'. This is a report projection, not a limit
on feature execution or on the complete native Feature. Ordinary numerical
and problem-result arrays are not truncated by this feature-metadata rule.
Long generated display labels use a bounded prefix and CRC-32 suffix in MATLAB; Python uses SHA-256 for that display suffix. These labels do not change seeds, scientific feature identity, or output-directory uniqueness. Equal shortened folder names across languages are not promised.
Feature option values are checked when a Feature is built. The numeric
options perturbation_level, noise_level, condition_factor,
mesh_size, nan_rate and significant_digits must be finite real
scalars within their documented bounds; MATLAB stores them as double, and an
integer-class value must be exactly representable as double.
perturbation_level is a nonnegative scalar: MATLAB has no coordinatewise
vector amplitudes. Logical options accept true/false or numeric 0/1 and
are stored as logical. Text choices such as mesh_type are lowercase and
case-sensitive. Option names are case-insensitive; two spellings of one name
(noise_level and NOISE_LEVEL) in one struct or name/value list are
rejected (MATLAB:Feature:DuplicateOption) because the configuration would
be ambiguous. Earlier versions accepted some values that could not describe a
valid experiment: NaN or infinite magnitudes, any perturbation_level
(including negative, text or vector values), and integer or single classes that
then computed in integer or single arithmetic. Saved Features are revalidated
when executed again, including through loadBenchmarkOptions. Invalid values
are rejected explicitly; supported integer and single scalars are instead
canonicalized to double, just like fresh input. benchmark with load still reanalyses
the saved histories, because it does not execute the saved Feature.
On the identity and single-stage strategies, FeaturedProblem now counts
nonlinear-constraint evaluations once per recorded query, as the composed
strategy and the Python package already did. Earlier versions reported the
number of constraint components while fewer queries had been recorded. They
could therefore serve stale constraint values once that number reached the
evaluation budget, and they passed repeated indices to stochastic constraint
modifiers. New executions record runtime_policy='matlab-legacy-single-v2'.
Archives written under matlab-legacy-single-v1 keep their recorded policy
when they are loaded. Numerical results are unchanged for constraint channels
with at most one component.
Composed features (two or more effective stages) record
seed_policy='matlab-stage-horner32-v2'. The per-stage, per-channel seeds
are derived exactly as under version 1 (an exact 32-bit Horner fold over the
run seed and the stage identity); version 2 seeds every per-query stream of a
composed view by folding the IEEE-754 words of the observed payload (values,
point, served index) with the same rule. Version 1 handed that payload to the
legacy product mixer, so a zero coordinate, a zero value or a zero counter
removed the dependence on the rest of the payload. Archives written under
version 1 keep their recorded policy string. The identity and single-stage
strategies are unchanged and keep their established legacy streams. The fold
is a finite 32-bit hash: it is not a statistical independence guarantee and
distinct payloads can still collide.
Migration note: is_stochastic is a read-only property of Feature (as
are name, stages and declared); the earlier method call syntax
is_stochastic(F) is no longer available because MATLAB cannot expose one
name as both a property and a method. Use F.is_stochastic.
Example 4: testing parametrized solvers#
(See also the file in the repository: matlab/examples/example4.m)
If you want to benchmark a solver with one variable parameter, you can define function handles by looping over the parameter values. For example, if solver accepts the signature @(fun, x0, para), and you want to benchmark it with the parameter para taking values from 1 to 3, you can run:
solvers = cell(1, 3);
options.solver_names = cell(1, 3);
for i = 1:3
solvers{i} = @(fun, x0) solver(fun, x0, i);
options.solver_names{i} = ['solver' num2str(i)];
end
scores = benchmark(solvers, options)
Example 5: customizing the test suite#
(See also the file in the repository: matlab/examples/example5.m)
OptiProfiler allows you to customize the test suite by creating your own feature and loading your own problem library. For example, if you want to create a new feature that adds noise to the objective function and perturbs the initial guess at the same time, you can try the following:
options.feature_name = 'custom';
options.mod_fun = @(x, rand_stream, problem) problem.fun(x) + 1e-3 * rand_stream.randn(1);
options.mod_x0 = @(rand_stream, problem) problem.x0 + 1e-3 * rand_stream.randn(problem.n, 1);
scores = benchmark({@solver1, @solver2}, options)
Installed providers are selected with options.plibs. See
MATLAB problem libraries for installing MatCUTEst and SOLAR, and
Custom MATLAB problem libraries to register a library stored in any
user-owned directory. The detailed provider-authoring contract is kept out of
this solver tutorial so that the benchmark workflow remains focused.
Example 6: wrapping solvers with nonlinear constraints#
(See also the file in the repository: matlab/examples/example6.m)
For nonlinearly constrained problems, OptiProfiler calls each solver with the signature
x = solver(fun, x0, xl, xu, aub, bub, aeq, beq, cub, ceq)
where cub(x) <= 0 contains the nonlinear inequality constraints and
ceq(x) = 0 contains the nonlinear equality constraints. MATLAB solvers
such as fmincon instead expect one nonlinear constraint callback
nonlcon returning both values. The small but important adapter is
deal: the expression @(x) deal(cub(x), ceq(x)) evaluates
OptiProfiler’s two callbacks and returns them as the two outputs expected by
fmincon, i.e., [c, ceq] = nonlcon(x). See the MathWorks
documentation for
fmincon nonlinear constraints
and deal.
function x = fmincon_short(fun, x0, xl, xu, aub, bub, aeq, beq, cub, ceq)
x = fmincon_wrapper(fun, x0, xl, xu, aub, bub, aeq, beq, cub, ceq, 100);
end
function x = fmincon_long(fun, x0, xl, xu, aub, bub, aeq, beq, cub, ceq)
x = fmincon_wrapper(fun, x0, xl, xu, aub, bub, aeq, beq, cub, ceq, 200);
end
function x = fmincon_wrapper(fun, x0, xl, xu, aub, bub, aeq, beq, cub, ceq, max_fun_evals)
% Convert OptiProfiler's separate nonlinear callbacks to fmincon's
% two-output callback: [c, ceq] = nonlcon(x).
nonlcon = @(x) deal(cub(x), ceq(x));
options = optimoptions('fmincon', 'MaxFunctionEvaluations', max_fun_evals);
x = fmincon(fun, x0, aub, bub, aeq, beq, xl, xu, nonlcon, options);
end
Then pass the wrappers to benchmark as ordinary solvers:
options.ptype = 'n';
options.problem_names = {'HS10', 'HS11', 'HS12'};
options.plibs = {'s2mpj'};
options.mindim = 2;
options.maxdim = 5;
options.max_eval_factor = 500;
options.draw_hist_plots = 'none';
options.n_jobs = 1;
options.solver_names = {'fmincon short', 'fmincon long'};
scores = benchmark({@fmincon_short, @fmincon_long}, options)