optiprofiler.benchmark

Contents

optiprofiler.benchmark#

optiprofiler.benchmark(solvers: list[Callable[[...], Any]] | None = None, /, **kwargs) → tuple[ndarray, ndarray | None, list[dict] | None][source]#

Benchmark optimization solvers on a set of problems with specified features.

This function creates multiple profiles for benchmarking optimization solvers on a set of problems with different features. It generates performance profiles, data profiles, and log-ratio profiles [1], [2], [4], [5] for the given solvers on various test suites, returning solver scores based on the profiles.

Parameters:
solverslist of callable, optional if load is provided

Solvers to benchmark. Each solver must be a callable accepting corresponding arguments depending on the test suite you choose:

  • for an unconstrained problem, solver(fun, x0) -> numpy.ndarray, shape (n,), where fun is the objective function accepting a 1-D array and returning a float, and x0 is the initial guess (1-D array);

  • for a bound-constrained problem, solver(fun, x0, xl, xu) -> numpy.ndarray, shape (n,), where xl and xu are the lower and upper bounds (1-D arrays, may contain -numpy.inf or numpy.inf);

  • for a linearly constrained problem, solver(fun, x0, xl, xu, aub, bub, aeq, beq) -> numpy.ndarray, shape (n,), where aub and aeq are the coefficient matrices of the linear inequality and equality constraints, and bub and beq are the right-hand side vectors;

  • for a nonlinearly constrained problem, solver(fun, x0, xl, xu, aub, bub, aeq, beq, cub, ceq) -> numpy.ndarray, shape (n,), where cub and ceq are the nonlinear inequality and equality constraint functions accepting a 1-D array and returning a 1-D array.

All vectors and matrices mentioned above are numpy.ndarray.

If the load option is provided, solvers can be None, in which case data from a previous experiment will be loaded to generate profiles.

Returns:
solver_scoresnumpy.ndarray

Scores of the solvers based on the profiles. See ‘score_fun’ in ‘Other Parameters’ for more details.

profile_scoresnumpy.ndarray or None

A 4D array containing scores for all profiles. The first dimension corresponds to the index of the solver, the second to the index of tolerance starting from 1, the third represents history-based or output-based profiles, and the fourth represents performance profiles, data profiles, or log-ratio profiles. A profile at a tolerance no solver meets scores 0.

curveslist of dict or None

A list containing the curves of all the profiles. A profile at a tolerance no solver meets keeps None in place of its curves.

Other Parameters:
Feature options:
feature_namestr, optional

Name of the feature to apply to problems. The available features are ‘plain’, ‘perturbed_x0’, ‘noisy’, ‘truncated’, ‘permuted’, ‘linearly_transformed’, ‘random_nan’, ‘unrelaxable_constraints’, ‘nonquantifiable_constraints’, ‘quantized’, and ‘custom’. Default is ‘plain’. Several features can be composed by joining their names with ‘+’, for example ‘noisy+truncated’: the first name is applied to the original problem first, so each objective value is made noisy and the noisy value is then truncated. Supplied feature options are passed to every stage that accepts them, ‘n_runs’ is global, and ‘plain’ stages are identities. See the user guide for the rules.

featuredict, list, tuple or Feature, optional

Structured specification of the feature: one stage entry {'name': ..., 'options': {...}}, an ordered list/tuple of entries (bare names allowed) or an already built Feature (used as it is, without reparsing). Each stage owns the options inside its entry, so repeated stages and different values of a shared key are expressible; n_runs stays a top-level option. feature and feature_name cannot both be given, flat stage options are rejected with feature, and feature cannot be combined with load.

n_runsint, optional

The number of runs of the experiments with the given feature. This is an experiment option, never a feature option: the experiment plan records it once, no stage carries a run count (it is rejected inside a feature entry and by Feature(..., n_runs=...)), and the plain reference of run_plain always uses one run. Unless given, it is 5 when solver_isrand marks a randomized solver (not in a load), and otherwise the established default of the feature: 5 for ‘perturbed_x0’, ‘noisy’ (1 when noise_mode='deterministic'), ‘permuted’, ‘linearly_transformed’ (also when rotated=False), ‘random_nan’ and ‘truncated’ with perturbed_trailing_digits=True; 1 for ‘plain’, ‘truncated’, ‘unrelaxable_constraints’, ‘nonquantifiable_constraints’, ‘quantized’ and ‘custom’ (although ‘custom’ counts as stochastic). A composition defaults to the largest default of its effective stages. A present None is invalid.

distributionstr or callable, optional

The distribution of perturbation in ‘perturbed_x0’ feature or random noise in ‘noisy’ feature. It should be either a str (or char), or a callable (random_stream, dimension) -> random vector, accepting a random_stream and the dimension of a problem and returning a random vector with the given dimension. In ‘perturbed_x0’ case, the str should be either ‘spherical’ or ‘gaussian’ (default is ‘spherical’). In ‘noisy’ case, the str should be either ‘gaussian’ or ‘uniform’ (default is ‘gaussian’), and the callable should accept a random stream and output size.

perturbation_levelfloat, optional

The magnitude of the perturbation to the initial guess in the ‘perturbed_x0’ feature. Default is 1e-3.

noise_levelfloat, optional

The magnitude of the noise in the ‘noisy’ feature. Default is 1e-3.

noise_typestr, optional

The type of the noise in the ‘noisy’ features. It should be either ‘absolute’, ‘relative’, or ‘mixed’. Default is ‘mixed’.

noise_modestr, optional

The mode of the noise in the ‘noisy’ feature. It should be either ‘random’ or ‘deterministic’. Default is ‘random’. When it is ‘deterministic’ and n_runs is not specified, n_runs defaults to 1.

noise_mapstr or callable, optional

The deterministic scalar noise map in the ‘noisy’ feature. It should be either ‘chebyshev’ or a callable x -> noise returning a real scalar. It is used only when noise_mode is ‘deterministic’. Default is ‘chebyshev’. The built-in ‘chebyshev’ map follows the deterministic noise model in Moré and Wild [5].

significant_digitsint, optional

The number of significant digits in the ‘truncated’ feature. Default is 6.

perturbed_trailing_digitsbool, optional

Whether we will randomize the trailing digits of the objective function value in the ‘truncated’ feature. Default is False.

rotatedbool, optional

Whether to use a random or given rotation matrix to rotate the coordinates of a problem in the ‘linearly_transformed’ feature. Default is True.

condition_factorfloat, optional

The scaling factor of the condition number of the linear transformation in the ‘linearly_transformed’ feature. More specifically, the condition number of the linear transformation will be 2**sqrt(condition_factor * n / 2) for dimension n >= 2 (it is 1 when n = 1). Default is 0.

nan_ratefloat, optional

The probability that the evaluation of the objective function will return np.nan in the ‘random_nan’ feature. Default is 0.05.

unrelaxable_boundsbool, optional

Whether the bound constraints are unrelaxable or not in the ‘unrelaxable_constraints’ feature. Default is True.

unrelaxable_linear_constraintsbool, optional

Whether the linear constraints are unrelaxable or not in the ‘unrelaxable_constraints’ feature. Default is False.

unrelaxable_nonlinear_constraintsbool, optional

Whether the nonlinear constraints are unrelaxable or not in the ‘unrelaxable_constraints’ feature. Default is False.

mesh_sizefloat, optional

The size of the mesh in the ‘quantized’ feature. Default is 1e-3.

mesh_typestr, optional

The type of the mesh in the ‘quantized’ feature. It should be either ‘absolute’ or ‘relative’. Default is ‘absolute’.

ground_truthbool, optional

Whether the featured problem is the ground truth or not in the ‘quantized’ feature. Default is True.

mod_x0callable, optional

The modifier function to modify the initial guess in the ‘custom’ feature. It should be a callable (random_stream, problem) -> modified_x0, where problem is an instance of the class Problem, and modified_x0 is the modified initial guess. No default.

mod_affinecallable, optional

The modifier function to generate the affine transformation applied to the variables in the ‘custom’ feature. It should be a callable (random_stream, problem) -> (A, b, inv), where problem is an instance of the class Problem, A is the matrix of the affine transformation, b is the vector of the affine transformation, and inv is a candidate inverse of A. No default. The callable is asked once per problem and seed, and the initial point, the bounds, the linear constraints and every evaluation use that one answer. The triple is validated when the problem is built, and an error is raised unless the arrays are real, finite and of matching sizes, norm(abs(inv) @ abs(A), inf) < 1 / eps, and both inverse residuals pass the finite-precision check norm(max(abs(A @ inv - I) - 64 * n * eps * abs(A) @ abs(inv), 0), 'fro') <= 1e-8 * n and the same for inv @ A. Here eps is double-precision machine epsilon; these matrix-test quantities must be finite. Independently, rows then columns of A are scaled by powers of two to form E without losing any entries, and cond(E, 1) < 1 / eps is required. These numerical checks are not a proof that inv is an exact inverse. Integer and extended precision inputs are rounded to double precision before validation and use. If A is exactly diagonal and diag(inv) is its reciprocal to roundoff, the bounds stay bounds; otherwise every finite bound is posed as a linear constraint, so an off-diagonal entry of A is never ignored, however small. Transport rejects finite bounds, coefficients or right-hand sides becoming non-finite, nonzero scaled bounds below the smallest normal magnitude (about 2.2e-308), and row or right-hand-side entries whose sum of absolute nonzero product terms lies below that magnitude. A strict original bound interval must not collapse to equal endpoints after translation or diagonal scaling. These are stage-local safeguards, not an exact transport certificate. Unless mod_x0 is supplied, a finite original x0 uses y = inv @ (x0 - b) unchanged only if y, its mapped point and the rounding allowance are finite, and abs(A @ y + b - x0) is at most both 64 * n * eps * (abs(A) @ abs(y) + abs(b) + abs(x0)) and sqrt(eps) * minimum(abs(x0), abs(x0 - b)), componentwise. There is no absolute floor: an error at a zero coordinate of either x0 or x0 - b triggers a solve. Both scales are checked so a large coordinate origin cannot hide error in a small centered value. If either check fails, A @ y = x0 - b is solved independently and the solved point is checked for finiteness and the rounding allowance only; failure raises an initial-point error. The extra local threshold triggers a solve instead of trusting the inverse candidate; it is not a bound on the final forward error. Even an honestly computed inverse may trigger a solve, for example when a custom rotation mixes small and large coordinates. Intrinsic conditioning or mixed coordinate scales can cause forward drift even after solving. fun_init and maxcv_init describe the posed start, not guaranteed recovery of an originally feasible point. Since mod_linear_ub and mod_linear_eq replace linear constraints, they are refused if they would remove generated bound rows, unless mod_bounds also takes ownership of the bounds. Derivatives follow the same map by the chain rule (see FeaturedProblem). Construction checks do not recompute initial points or histories in previously saved experiments or built-object pickles; loading is not a migration of old results.

mod_boundscallable, optional

The modifier function to modify the bound constraints in the ‘custom’ feature. It should be a callable (random_stream, problem) -> (modified_xl, modified_xu), where problem is an instance of the class Problem, modified_xl is the modified lower bound, and modified_xu is the modified upper bound. No default. A supplied modifier replaces its own component verbatim, and mod_bounds owns the logical original box in every representation. When mod_affine uses generated bound rows, the framework therefore does not add that original box again as linear rows; explicit linear constraints are still transported. The truth always scores the bounds of the problem, and the reference of the problem is unknown after any mod_bounds.

mod_linear_ubcallable, optional

The modifier function to modify the linear inequality constraints in the ‘custom’ feature. It should be a callable (random_stream, problem) -> (modified_aub, modified_bub), where problem is an instance of the class Problem, modified_aub is the modified matrix of the linear inequality constraints, and modified_bub is the modified vector of the linear inequality constraints. No default.

mod_linear_eqcallable, optional

The modifier function to modify the linear equality constraints in the ‘custom’ feature. It should be a callable (random_stream, problem) -> (modified_aeq, modified_beq), where problem is an instance of the class Problem, modified_aeq is the modified matrix of the linear equality constraints, and modified_beq is the modified vector of the linear equality constraints. No default.

mod_funcallable, optional

The modifier function to modify the objective function in the ‘custom’ feature. It should be a callable (x, random_stream, problem) -> modified_fun, where x is the evaluation point, problem is an instance of the class Problem, and modified_fun is the modified objective function value. No default.

mod_cubcallable, optional

The modifier function to modify the nonlinear inequality constraints in the ‘custom’ feature. It should be a callable (x, random_stream, problem) -> modified_cub, where x is the evaluation point, problem is an instance of the class Problem, and modified_cub is the modified vector of the nonlinear inequality constraints. No default.

mod_ceqcallable, optional

The modifier function to modify the nonlinear equality constraints in the ‘custom’ feature. It should be a callable (x, random_stream, problem) -> modified_ceq, where x is the evaluation point, problem is an instance of the class Problem, and modified_ceq is the modified vector of the nonlinear equality constraints. No default.

Profile and plot options:
bar_colorslist or numpy.ndarray, optional

Two different colors for the bars of two solvers in the log-ratio profiles. It can be a list of short names of colors (‘r’, ‘g’, ‘b’, ‘c’, ‘m’, ‘y’, ‘k’) or a 2-by-3 array with each row being a RGB triplet. Default is set to the ‘line_colors’ option. The bars cycle through the given colors, so with a single color both solvers’ bars use it.

benchmark_idstr, optional

The identifier of the test. It is used to create the specific directory to store the results. Default is ‘out’ if the option ‘load’ is not provided, otherwise default is ‘.’.

draw_hist_plotsstr, optional

Whether or how to draw the history plots of all the problems. It can be either ‘none’, ‘sequential’, or ‘parallel’. If it is ‘none’, we will not draw the history plots. If it is ‘parallel’, we will draw the history plots at the same time when solvers are solving the problems. If it is ‘sequential’, we will draw the history plots after all the problems are solved. For each problem, the original combined history plot is saved as PROBLEM.pdf; raw-only and cumulative-minimum-only plots are saved under the raw and cummin subfolders. Default is ‘parallel’.

errorbar_typestr, optional

The type of the uncertainty interval that can be either ‘minmax’ or ‘meanstd’. When ‘n_runs’ is greater than 1, we run several times of the experiments and get average curves and uncertainty intervals. Default is ‘minmax’, meaning that we takes the pointwise minimum and maximum of the curves.

feature_stampstr, optional

The stamp of the feature with the given options. It is used to create the specific directory to store the results. Default depends on features.

hist_aggregationstr, optional

The aggregation method we use to reduce the number of points in the history plots. It can be ‘min’, ‘mean’, or ‘max’. Default is ‘min’.

line_colorslist, optional

The colors of the lines in the plots. It can be a list of any valid matplotlib colors (short names, hex strings, RGB tuples, etc.). Default line colors are from the matplotlib tab10 color cycle. Note that if the number of solvers is greater than the number of colors, we will cycle through the colors.

line_styleslist, optional

The Matplotlib line styles, such as ‘-’, ‘–’, ‘-.’, ‘:’, their names (e.g., ‘solid’ or ‘dashed’), or dash tuples (offset, dash_sequence). Color/marker format strings such as ‘r–’ or ‘-o’ are not line styles. Default line style order is [‘-’, ‘-.’, ‘–’, ‘:’]. Note that if the number of solvers is greater than the number of line styles, we will cycle through the styles.

line_widthsfloat or list, optional

The widths of the lines in the plots. It should be a positive float or a list. Default is 1.5. Note that if the number of solvers is greater than the number of line widths, we will cycle through the widths.

loadstr, optional

Loading the stored data from a completed experiment and draw profiles. It can be either ‘latest’ or a time stamp of an experiment in the format of ‘yyyyMMdd_HHmmss’, optionally followed by a collision suffix such as ‘_001’. No default. Note that if solvers is None, this key must be provided to load data from a previous experiment and generate profiles. The experiment is searched for in the current working directory joined with benchmark_id (up to five levels of subdirectories), not in savepath. Load only archives from trusted sources: saved Python options may contain pickle objects, whose deserialization can execute code.

max_eval_factorint, optional

The factor multiplied to each problem’s dimension to get the maximum number of evaluations for each problem. Default is 500.

max_tol_orderint, optional

The maximum order of the tolerance. In any profile (performance profiles, data profiles, and log-ratio profiles), we need to set a group of ‘tolerances’ to define the convergence test of the solvers. (Details can be found in the references.) We will set the tolerances as 10**(-k) for k = 1, 2, ..., max_tol_order. Default is 10.

merit_funcallable, optional

The merit function to measure the quality of a point using the objective function value and the maximum constraint violation. It should be a callable (fun_value, maxcv_value, maxcv_init) -> merit_value, where fun_value is the objective function value, maxcv_value is the maximum constraint violation, and maxcv_init is the maximum constraint violation at the initial guess. The default merit function varphi(x) is defined by the objective function f(x) and the maximum constraint violation v(x) as:

varphi(x) = f(x)                        if v(x) <= v1
varphi(x) = f(x) + 1e5 * (v(x) - v1)   if v1 < v(x) <= v2
varphi(x) = np.inf                       if v(x) > v2

where v1 = min(0.01, 1e-10 * max(1, v0)), v2 = max(0.1, 2 * v0), and v0 is the maximum constraint violation at the initial guess. NaN objective or constraint values are invalid evaluations, including when a custom merit function returns a finite value for them. A run with an undefined initial value, including a merit function value of NaN at the initial guess, is not declared passing. Otherwise, if varphi(x_0) is +inf, the existing degenerate-test convention accepts valid evaluations; if it is -inf, only a merit value of -inf passes. Both cases are listed in test_log/report.txt. In output-based profiles, an output obtained without any evaluation counts as one evaluation. A run whose solver raised an exception or returned nothing has no output: the initial point is recorded in its place, the run is listed in test_log/report.txt and it never passes the output-based test, whatever the target, while its history-based results stand. A solver that returns the initial point itself is not such a run. With problem, the score of a run whose initial merit is not finite is NaN (the relative decrease is undefined), whereas the profiles above apply the +inf convention.

n_jobsint, optional

The number of parallel jobs to run the test. Default is a conservative number of workers, chosen as about half of the available workers (at least 2 when more than one worker is available).

normalized_scoresbool, optional

Whether to normalize the scores of the solvers by the maximum score of the solvers. Default is True.

project_x0bool, optional

Whether to project the initial point to the feasible set. Default is False.

run_plainbool, optional

Whether to run an extra experiment with the ‘plain’ feature. Default is False.

savepathstr, optional

The path to store the results. Default is the current working directory. It does not affect where load searches.

score_funcallable, optional

The scoring function to calculate the scores of the solvers. It should be a callable profile_scores -> solver_scores, where profile_scores is a 4D array containing scores for all profiles. The first dimension of profile_scores corresponds to the index of the solver, the second corresponds to the index of tolerance starting from 1, the third represents history-based or output-based profiles, and the fourth represents performance profiles, data profiles, or log-ratio profiles. A history-based or output-based profile at a tolerance no solver meets has no curve and scores 0 for every solver. It must return one real score per solver: a vector of length n_solvers (a column or row array with n_solvers entries is flattened to that vector; any other shape raises an error). The default scoring function takes the average of the history-based performance profiles under all the tolerances.

score_onlybool, optional

Whether to only calculate the scores of the solvers without drawing the profiles and saving the data. Messages are printed to the console as in a saved run (subject to silent), but no log file is written. Default is False.

plot_optionsdict, optional

Native Matplotlib property mappings for figure, axes, xaxis, yaxis, xlabel, ylabel, title, text, line and legend; also accepts tick_params and a local rc mapping. For example, {'xlabel': {'fontsize': 18}, 'legend': {'fontsize': 14}}. Properties override package defaults for profiles, summaries and histories, including on load, without changing scores or source data. Legend entries remain solver-owned. Backend/interactive selection is not a style setting. Values must support serialization for parallel history rendering. Default is an empty mapping (existing style).

report_pathstr or pathlib.Path, optional

Write a compact, controller-private eval_report JSON and an adjacent <report-stem>.plot_data.json numerical companion. Neither path may already exist. The main file holds exact scalar facts, existing scores, and references; the companion holds bounded history summaries and full prepared plot coordinates/bands. Verify its SHA256 and evaluation ID before following references. Reporting does not change returned values, scoring, seeds, or plotting options. With score_only=True only these two explicitly requested JSON files are written; raw archives and PDFs are not enabled. Numerical, scoring, persistence and rendering outcomes remain separate. Neither file is automatically a public agent prompt. Default is no report.

score_weight_funcallable, optional

The weight function to calculate the scores of the solvers in the performance and data profiles. It should be a callable representing a nonnegative function in R^+. Default is a constant function returning 1.

seedint, optional

The seed of the random number generator. Default is 0.

semilogxbool, optional

Whether to use the semilogx scale during plotting profiles (performance profiles and data profiles). For data profiles, the plotted coordinate is log2(1 + n_eval / (n + 1)) but tick labels are displayed in the original n_eval / (n + 1) units, typically 0, 1, 2, 4, .... Default is True.

silentbool, optional

Whether to show the information of the progress. With True, no progress information is printed, also when loading saved results, and ‘solver_verbose’ is 0. Default is False.

solver_israndlist of bool, optional

Whether the solvers are randomized or not. Default is a list of bools of the same length as the number of solvers, where the value is True if the solver is randomized, and False otherwise. Note that if ‘n_runs’ is not specified, we will set it 5 for the randomized solvers.

solver_nameslist of str, optional

The names of the solvers. Default is the names of the callables in solvers.

solver_verboseint, optional

The level of the verbosity of the solvers. 0 means no verbosity, 1 means some verbosity, and 2 means full verbosity. With ‘silent’ True the level is 0, so nothing of a solver is printed, and a level of 1 or 2 is replaced with a warning; a failing solver is still recorded in report.txt and the machine-readable report. Default is 1.

solvers_to_loadlist of int, optional

The indices of the solvers to load when the ‘load’ option is provided. It can be a list of different integers selected from 0 to the total number of solvers minus 1 of the loading experiment. At least two indices should be provided. Default is all the solvers.

summarize_data_profilesbool, optional

Whether to add all the data profiles to the summary PDF. Default is True.

summarize_log_ratio_profilesbool, optional

Whether to add all the log-ratio profiles to the summary PDF. Default is False.

summarize_output_based_profilesbool, optional

Whether to add all the output-based profiles of the selected profiles to the summary PDF. Default is True.

summarize_performance_profilesbool, optional

Whether to add all the performance profiles to the summary PDF. Default is True.

xlabel_data_profilestr, optional

The label of the x-axis of the data profiles. Default is ‘Number of simplex gradients’. Note: LaTeX formatting is supported. The same applies to the options ‘xlabel_log_ratio_profile’, ‘xlabel_performance_profile’, ‘ylabel_data_profile’, ‘ylabel_log_ratio_profile’, and ‘ylabel_performance_profile’. The text of the x-axis labels is passed to the renderer verbatim. Under LaTeX, write ‘%’ for a percent sign: a bare ‘%’ starts a LaTeX comment and cuts the label. Without LaTeX (see ‘text.usetex’ in the ‘rc’ group of ‘plot_options’), ‘%’ keeps its backslash; ‘$50%$’ works in both cases.

xlabel_log_ratio_profilestr, optional

The label of the x-axis of the log-ratio profiles. Default is ‘Problem’.

xlabel_performance_profilestr, optional

The label of the x-axis of the performance profiles. Default is ‘Performance ratio’.

ylabel_data_profilestr, optional

The label of the y-axis of the data profiles. Default is ‘Data profiles ($mathrm{tol} = %s$)’, where ‘%s’ will be replaced by the current tolerance in LaTeX format. You can also use ‘%s’ in your custom label, and it will be replaced accordingly. Keep ‘%s’ inside ‘$…$’, as the defaults do, because the tolerance is LaTeX math: outside math mode it is literal text such as ‘10^{-1}’ without LaTeX, and it raises a LaTeX error under LaTeX. Only ‘%s’ is a placeholder: ‘%%’ and any other ‘%’ give a literal percent sign, escaped automatically where LaTeX or a ‘$…$’ segment needs it (so ‘50% solved at $%s$’ is a valid label). The same applies to the options ‘ylabel_log_ratio_profile’ and ‘ylabel_performance_profile’.

ylabel_log_ratio_profilestr, optional

The label of the y-axis of the log-ratio profiles. Default is ‘Log-ratio profiles ($mathrm{tol} = %s$)’, where ‘%s’ will be replaced by the current tolerance in LaTeX format.

ylabel_performance_profilestr, optional

The label of the y-axis of the performance profiles. Default is ‘Performance profiles ($mathrm{tol} = %s$)’, where ‘%s’ will be replaced by the current tolerance in LaTeX format.

Problem options:
Options in this part are used to select problems for benchmarking.
First select which problem libraries to use based on the ‘plibs’
option. Then select problems from these libraries according to the
given options (‘problem_names’, ‘ptype’, ‘mindim’, ‘maxdim’, ‘minb’,
‘maxb’, ‘minlcon’, ‘maxlcon’, ‘minnlcon’, ‘maxnlcon’, ‘mincon’,
‘maxcon’, and ‘excludelist’). Library-specific selection and loading
behavior is configured separately through ‘plib_options’.
plibslist of str, optional

The problem libraries to be used. It should be a list of strs. Each name must be an ASCII Python identifier: it must start with a letter or underscore and contain only letters, digits, and underscores. The built-in choices are 's2mpj' and 'custom'. Default setting is 's2mpj'. Separately installed problem-library packages, including the OptiProfiler PyCUTEst, SOLAR, and experimental RS13 providers, are discovered through the optiprofiler.problem_libraries entry-point group. You can also use an unpackaged local library by specifying its name here together with the custom_problem_libs_path option.

plib_optionsmapping, optional

Per-experiment options grouped by problem-library name. Each value must be a mapping understood and validated by that library. For example, a hypothetical API-v1 plugin may accept {'myproblems': {'variant': 'large'}} without changing another library or process-wide defaults. Keys for libraries not selected in plibs are rejected. Per-experiment values take precedence over set_plib_config(), environment-level settings, and package-owned defaults. The validated effective mapping, including defaults, is saved in test_log/options_refined.pkl and in each library’s data in test_log/data_for_loading.h5. This option is rejected with problem and load, because those modes do not invoke a problem library. Default is an empty mapping.

ptypestr, optional

The type of the problems to be selected. It should be a str consisting of any combination of ‘u’ (unconstrained), ‘b’ (bound constrained), ‘l’ (linearly constrained), and ‘n’ (nonlinearly constrained), such as ‘b’, ‘ul’, ‘ubn’. Default is ‘u’.

mindimint, optional

The minimum dimension of the problems to be selected. It must be a finite integer greater than or equal to 1. Default is 1.

maxdimint or positive infinity, optional

The maximum dimension of the problems to be selected. It must be an integer greater than or equal to 1, or positive infinity. A positive infinity (np.inf) means that no maximum dimension is imposed. Default is mindim + 1.

minbint, optional

The minimum number of bound constraints of the problems to be selected. It must be a finite integer greater than or equal to 0. Default is 0.

maxbint or positive infinity, optional

The maximum number of bound constraints of the problems to be selected. It must be an integer greater than or equal to 0, or positive infinity. A positive infinity (np.inf) means that no maximum number of bound constraints is imposed. Default is minb + 10.

minlconint, optional

The minimum number of linear constraints of the problems to be selected. It must be a finite integer greater than or equal to 0. Default is 0.

maxlconint or positive infinity, optional

The maximum number of linear constraints of the problems to be selected. It must be an integer greater than or equal to 0, or positive infinity. A positive infinity (np.inf) means that no maximum number of linear constraints is imposed. Default is minlcon + 10.

minnlconint, optional

The minimum number of nonlinear constraints of the problems to be selected. It must be a finite integer greater than or equal to 0. Default is 0.

maxnlconint or positive infinity, optional

The maximum number of nonlinear constraints of the problems to be selected. It must be an integer greater than or equal to 0, or positive infinity. A positive infinity (np.inf) means that no maximum number of nonlinear constraints is imposed. Default is minnlcon + 10.

minconint, optional

The minimum number of linear and nonlinear constraints of the problems to be selected. It must be a finite integer greater than or equal to 0. Default is min(minlcon, minnlcon).

maxconint or positive infinity, optional

The maximum number of linear and nonlinear constraints of the problems to be selected. It must be an integer greater than or equal to 0, or positive infinity. A positive infinity (np.inf) means that no maximum number of constraints is imposed. Default is max(maxlcon, maxnlcon).

custom_problem_libs_pathstr or Path, optional

The path to a directory containing custom problem libraries, or a direct path to one custom problem library. Each custom problem library must contain a tools module named ‘<library_name>_tools.py’ with two functions: ‘<library_name>_load’ and ‘<library_name>_select’. This option allows users to use their own problem libraries without modifying the installed package. If an explicitly supplied custom library has the same name as another provider, the custom library is used for that run. Without an explicit custom path, duplicate providers are rejected as ambiguous, except that Python 3.8 and 3.9 cannot tell two distributions apart that declare the identical entry point, which therefore resolves. Default is None.

excludelistlist, optional

The list of problems to be excluded. Default is not to exclude any problem.

problem_nameslist of str, optional

The names of the problems to be selected. It should be a list of strs. Default is not to select any problem by name but by the options above.

problemProblem, optional

A problem to be benchmarked. It should be an instance of the class Problem. If it is provided, we will only run the test on this problem with the given feature and draw the history plots. It cannot be combined with load. Default is not to set any problem.

Raises:
TypeError

If an argument received an invalid value.

ValueError

If the arguments are inconsistent.

See also

Problem

Representation of optimization problems.

Feature

Feature applied to problems during benchmarking.

FeaturedProblem

Problem equipped with a specific feature.

Notes

The current version supports benchmarking derivative-free optimization solvers.

Caution

The log-ratio profiles are available only when there are exactly two solvers. For more information on performance and data profiles, see [1], [2], [5]. For that of log-ratio profiles, see [4], [6].

Caution

All callable arguments (solvers, distribution, noise_map, mod_x0, mod_affine, mod_bounds, mod_linear_ub, mod_linear_eq, mod_fun, mod_cub, mod_ceq, merit_fun, score_fun, score_weight_fun) must be picklable for parallel execution (n_jobs > 1). In particular, lambda functions are not picklable and will cause the benchmark to fall back to sequential mode automatically. To take advantage of parallel execution, define named functions (using def) instead of lambda expressions.

  1. S2MPJ (see [3]) is the bundled default problem library. PyCUTEst, SOLAR (see [7]), and RS13 providers are distributed and tested independently; install a compatible provider package before selecting its public name. OptiProfiler discovers these packages through the versioned problem-library protocol. For local development without packaging, see the custom_problem_libs_path option or the guide on our website. An explicitly supplied custom library overrides other providers with the same name for that run; all other duplicate providers are rejected, except that Python 3.8 and 3.9 cannot tell two distributions apart that declare the identical entry point, which therefore resolves.

  2. Problem libraries may define different library-specific options. For a reproducible run, provide them through plib_options. Package-owned defaults may come from config.txt or another plugin-specific source. set_plib_config() remains available for process-level defaults, and some adapters also support environment variables such as S2MPJ_VARIABLE_SIZE. The precedence is plib_options first, then set_plib_config(), then the library-owned environment/configuration and defaults. See get_plib_config() for the current process-level values.

  3. When the load option is provided, the function loads data from a previous experiment and draws profiles using the provided options. Available options in this mode are:

    • Profile and plot options: benchmark_id, solver_names, feature_stamp, errorbar_type, savepath, max_tol_order, merit_fun, run_plain, score_only, summarize_performance_profiles, summarize_data_profiles, summarize_log_ratio_profiles, summarize_output_based_profiles, silent, semilogx, normalized_scores, score_weight_fun, score_fun, solvers_to_load, line_colors, line_styles, line_widths, bar_colors.

    • Feature options: none.

    • Problem options: plibs, ptype, mindim, maxdim, minb, maxb, minlcon, maxlcon, minnlcon, maxnlcon, mincon, maxcon, excludelist.

  4. More information about OptiProfiler can be found at https://www.optprof.com.

References

[1] (1,2)

E. D. Dolan and J. J. Moré. Benchmarking optimization software with performance profiles. Math. Program., 91(2):201–213, 2002. doi:10.1007/s101070100263 <https://doi.org/10.1007/s101070100263>.

[2] (1,2)

N. Gould and J. Scott. A note on performance profiles for benchmarking software. ACM Trans. Math. Software, 43(2):15:1–5, 2016. doi:10.1145/2950048 <https://doi.org/10.1145/2950048>.

[3]

S. Gratton and Ph. L. Toint. S2MPJ and CUTEst optimization problems for Matlab, Python and Julia. arXiv:2407.07812, 2024.

[4] (1,2)

J. L. Morales. A numerical study of limited memory BFGS methods. Appl. Math. Lett., 15(4):481–487, 2002. doi:10.1016/S0893-9659(01)00162-8 <https://doi.org/10.1016/S0893-9659(01)00162-8>.

[5] (1,2,3)

J. J. Moré and S. M. Wild. Benchmarking derivative-free optimization algorithms. SIAM J. Optim., 20(1):172–191, 2009. doi:10.1137/080724083 <https://doi.org/10.1137/080724083>.

[6]

H.-J. M. Shi, M. Q. Xuan, F. Oztoprak, and J. Nocedal. On the numerical performance of finite-difference-based methods for derivative-free optimization. Optim. Methods Softw., 38(2):289–311, 2023. doi:10.1080/10556788.2022.2121832 <https://doi.org/10.1080/10556788.2022.2121832>.

[7]

N. Andrés-Thió, C. Audet, M. Diago, A. E. Gheribi, S. Le Digabel, X. Lebeuf, M. Lemyre-Garneau, and C. Tribes. solar: A solar thermal power plant simulator for blackbox optimization benchmarking. Optimization and Engineering, 2025. doi:10.1007/s11081-024-09952-x <https://doi.org/10.1007/s11081-024-09952-x>.

Examples

Benchmark two toy solvers that randomly sample around the initial point. Save the following as a .py file and run it. The if __name__ == '__main__' guard is required on every platform when n_jobs > 1 (the default on a multi-core machine): parallel workers are started with the spawn method, which imports the script again in each worker. A call reached while a worker imports the script raises a RuntimeError instead of running the benchmark again.

import numpy as np
from optiprofiler import benchmark

def solver1(fun, x0):
    rng = np.random.default_rng(0)
    best_x, best_f = x0, fun(x0)
    for _ in range(10 * len(x0)):
        x = x0 + rng.standard_normal(len(x0))
        if fun(x) < best_f:
            best_x, best_f = x, fun(x)
    return best_x

def solver2(fun, x0):
    rng = np.random.default_rng(0)
    best_x, best_f = x0, fun(x0)
    for _ in range(20 * len(x0)):
        x = x0 + rng.standard_normal(len(x0))
        if fun(x) < best_f:
            best_x, best_f = x, fun(x)
    return best_x

if __name__ == '__main__':
    scores = benchmark([solver1, solver2])