optiprofiler.benchmark

optiprofiler.benchmark(solvers: list[callable] | None = None, /, **kwargs) → tuple[ndarray, ndarray | None, list[dict] | None]

Benchmark optimization solvers on a set of problems with specified features.

This function creates multiple profiles for benchmarking optimization solvers on a set of problems with different features. It generates performance profiles, data profiles, and log-ratio profiles [1], [2], [4], [5] for the given solvers on various test suites, returning solver scores based on the profiles.

Parameters:

solverslist of callable, optional if ‘load’ in **kwargs

Solvers to benchmark. Each solver must be a callable accepting corresponding arguments depending on the test suite you choose:

• for an unconstrained problem, solver(fun, x0) -> numpy.ndarray, shape (n,), where fun is the objective function accepting a 1-D array and returning a float, and x0 is the initial guess (1-D array);

• for a bound-constrained problem, solver(fun, x0, xl, xu) -> numpy.ndarray, shape (n,), where xl and xu are the lower and upper bounds (1-D arrays, may contain -numpy.inf or numpy.inf);

• for a linearly constrained problem, solver(fun, x0, xl, xu, aub, bub, aeq, beq) -> numpy.ndarray, shape (n,), where aub and aeq are the coefficient matrices of the linear inequality and equality constraints, and bub and beq are the right-hand side vectors;

• for a nonlinearly constrained problem, solver(fun, x0, xl, xu, aub, bub, aeq, beq, cub, ceq) -> numpy.ndarray, shape (n,), where cub and ceq are the nonlinear inequality and equality constraint functions accepting a 1-D array and returning a 1-D array.

All vectors and matrices mentioned above are numpy.ndarray.

If the ‘load’ option is provided in **kwargs, solvers can be None, in which case data from a previous experiment will be loaded to generate profiles.

Returns:

solver_scoresnumpy.ndarray

Scores of the solvers based on the profiles. See ‘score_fun’ in ‘Other Parameters’ for more details.

profile_scoresnumpy.ndarray or None

A 4D array containing scores for all profiles. The first dimension corresponds to the index of the solver, the second to the index of tolerance starting from 1, the third represents history-based or output-based profiles, and the fourth represents performance profiles, data profiles, or log-ratio profiles.

curveslist of dict or None

A list containing the curves of all the profiles.

Other Parameters:

*Options for features:*

feature_namestr, optional

Name of the feature to apply to problems. The available features are ‘plain’, ‘perturbed_x0’, ‘noisy’, ‘truncated’, ‘permuted’, ‘linearly_transformed’, ‘random_nan’, ‘unrelaxable_constraints’, ‘nonquantifiable_constraints’, ‘quantized’, and ‘custom’. Default is ‘plain’.

n_runsint, optional

The number of runs of the experiments with the given feature. Default is 5 for stochastic features and 1 for deterministic features.

distributionstr or callable, optional

The distribution of perturbation in ‘perturbed_x0’ feature or noise in ‘noisy’ feature. It should be either a str (or char), or a callable (random_stream, dimension) -> random vector, accepting a random_stream and the dimension of a problem and returning a random vector with the given dimension. In ‘perturbed_x0’ case, the str should be either ‘spherical’ or ‘gaussian’ (default is ‘spherical’). In ‘noisy’ case, the str should be either ‘gaussian’ or ‘uniform’ (default is ‘gaussian’).

perturbation_levelfloat, optional

The magnitude of the perturbation to the initial guess in the ‘perturbed_x0’ feature. Default is 1e-3.

noise_levelfloat, optional

The magnitude of the noise in the ‘noisy’ feature. Default is 1e-3.

noise_typestr, optional

The type of the noise in the ‘noisy’ features. It should be either ‘absolute’, ‘relative’, or ‘mixed’. Default is ‘mixed’.

significant_digitsint, optional

The number of significant digits in the ‘truncated’ feature. Default is 6.

perturbed_trailing_digitsbool, optional

Whether we will randomize the trailing digits of the objective function value in the ‘truncated’ feature. Default is False.

rotatedbool, optional

Whether to use a random or given rotation matrix to rotate the coordinates of a problem in the ‘linearly_transformed’ feature. Default is True.

condition_factorfloat, optional

The scaling factor of the condition number of the linear transformation in the ‘linearly_transformed’ feature. More specifically, the condition number of the linear transformation will be 2 ** (condition_factor * n / 2), where n is the dimension of the problem. Default is 0.

nan_ratefloat, optional

The probability that the evaluation of the objective function will return np.nan in the ‘random_nan’ feature. Default is 0.05.

unrelaxable_boundsbool, optional

Whether the bound constraints are unrelaxable or not in the ‘unrelaxable_constraints’ feature. Default is True.

unrelaxable_linear_constraintsbool, optional

Whether the linear constraints are unrelaxable or not in the ‘unrelaxable_constraints’ feature. Default is False.

unrelaxable_nonlinear_constraintsbool, optional

Whether the nonlinear constraints are unrelaxable or not in the ‘unrelaxable_constraints’ feature. Default is False.

mesh_sizefloat, optional

The size of the mesh in the ‘quantized’ feature. Default is 1e-3.

mesh_typestr, optional

The type of the mesh in the ‘quantized’ feature. It should be either ‘absolute’ or ‘relative’. Default is ‘absolute’.

ground_truthbool, optional

Whether the featured problem is the ground truth or not in the ‘quantized’ feature. Default is True.

mod_x0callable, optional

The modifier function to modify the initial guess in the ‘custom’ feature. It should be a callable (random_stream, problem) -> modified_x0, where problem is an instance of the class Problem, and modified_x0 is the modified initial guess. No default.

mod_affinecallable, optional

The modifier function to generate the affine transformation applied to the variables in the ‘custom’ feature. It should be a callable (random_stream, problem) -> (A, b, inv), where problem is an instance of the class Problem, A is the matrix of the affine transformation, b is the vector of the affine transformation, and inv is the inverse of matrix A. No default.

mod_boundscallable, optional

The modifier function to modify the bound constraints in the ‘custom’ feature. It should be a callable (random_stream, problem) -> (modified_xl, modified_xu), where problem is an instance of the class Problem, modified_xl is the modified lower bound, and modified_xu is the modified upper bound. No default.

mod_linear_ubcallable, optional

The modifier function to modify the linear inequality constraints in the ‘custom’ feature. It should be a callable (random_stream, problem) -> (modified_aub, modified_bub), where problem is an instance of the class Problem, modified_aub is the modified matrix of the linear inequality constraints, and modified_bub is the modified vector of the linear inequality constraints. No default.

mod_linear_eqcallable, optional

The modifier function to modify the linear equality constraints in the ‘custom’ feature. It should be a callable (random_stream, problem) -> (modified_aeq, modified_beq), where problem is an instance of the class Problem, modified_aeq is the modified matrix of the linear equality constraints, and modified_beq is the modified vector of the linear equality constraints. No default.

mod_funcallable, optional

The modifier function to modify the objective function in the ‘custom’ feature. It should be a callable (x, random_stream, problem) -> modified_fun, where x is the evaluation point, problem is an instance of the class Problem, and modified_fun is the modified objective function value. No default.

mod_cubcallable, optional

The modifier function to modify the nonlinear inequality constraints in the ‘custom’ feature. It should be a callable (x, random_stream, problem) -> modified_cub, where x is the evaluation point, problem is an instance of the class Problem, and modified_cub is the modified vector of the nonlinear inequality constraints. No default.

mod_ceqcallable, optional

The modifier function to modify the nonlinear equality constraints in the ‘custom’ feature. It should be a callable (x, random_stream, problem) -> modified_ceq, where x is the evaluation point, problem is an instance of the class Problem, and modified_ceq is the modified vector of the nonlinear equality constraints. No default.

*Options for profiles and plots:*

bar_colorslist or numpy.ndarray, optional

Two different colors for the bars of two solvers in the log-ratio profiles. It can be a list of short names of colors (‘r’, ‘g’, ‘b’, ‘c’, ‘m’, ‘y’, ‘k’) or a 2-by-3 array with each row being a RGB triplet. Default is set to the first two colors in the ‘line_colors’ option.

benchmark_idstr, optional

The identifier of the test. It is used to create the specific directory to store the results. Default is ‘out’ if the option ‘load’ is not provided, otherwise default is ‘.’.

draw_hist_plotsstr, optional

Whether or how to draw the history plots of all the problems. It can be either ‘none’, ‘sequential’, or ‘parallel’. If it is ‘none’, we will not draw the history plots. If it is ‘parallel’, we will draw the history plots at the same time when solvers are solving the problems. If it is ‘sequential’, we will draw the history plots after all the problems are solved. Default is ‘parallel’.

errorbar_typestr, optional

The type of the uncertainty interval that can be either ‘minmax’ or ‘meanstd’. When ‘n_runs’ is greater than 1, we run several times of the experiments and get average curves and uncertainty intervals. Default is ‘minmax’, meaning that we takes the pointwise minimum and maximum of the curves.

feature_stampstr, optional

The stamp of the feature with the given options. It is used to create the specific directory to store the results. Default depends on features.

hist_aggregationstr, optional

The aggregation method we use to reduce the number of points in the history plots. It can be ‘min’, ‘mean’, or ‘max’. Default is ‘min’.

line_colorslist, optional

The colors of the lines in the plots. It can be a list of any valid matplotlib colors (short names, hex strings, RGB tuples, etc.). Default line colors are from the matplotlib tab10 color cycle. Note that if the number of solvers is greater than the number of colors, we will cycle through the colors.

line_styleslist of str, optional

The styles of the lines in the plots. It can be a list of strs that are the combinations of line styles (‘-’, ‘-.’, ‘–’, ‘:’) and markers (‘o’, ‘+’, ‘*’, ‘.’, ‘x’, ‘s’, ‘d’, ‘^’, ‘v’, ‘>’, ‘<’, ‘p’, ‘h’). Default line style order is [‘-’, ‘-.’, ‘–‘, ‘:’]. Note that if the number of solvers is greater than the number of line styles, we will cycle through the styles.

line_widthsfloat or list, optional

The widths of the lines in the plots. It should be a positive float or a list. Default is 1.5. Note that if the number of solvers is greater than the number of line widths, we will cycle through the widths.

loadstr, optional

Loading the stored data from a completed experiment and draw profiles. It can be either ‘latest’ or a time stamp of an experiment in the format of ‘yyyyMMdd_HHmmss’. No default. Note that if solvers is None, this key must be provided to load data from a previous experiment and generate profiles.

max_eval_factorint, optional

The factor multiplied to each problem’s dimension to get the maximum number of evaluations for each problem. Default is 500.

max_tol_orderint, optional

The maximum order of the tolerance. In any profile (performance profiles, data profiles, and log-ratio profiles), we need to set a group of ‘tolerances’ to define the convergence test of the solvers. (Details can be found in the references.) We will set the tolerances as 10**(-k) for k = 1, 2, ..., max_tol_order. Default is 10.

merit_funcallable, optional

The merit function to measure the quality of a point using the objective function value and the maximum constraint violation. It should be a callable (fun_value, maxcv_value, maxcv_init) -> merit_value, where fun_value is the objective function value, maxcv_value is the maximum constraint violation, and maxcv_init is the maximum constraint violation at the initial guess. The default merit function varphi(x) is defined by the objective function f(x) and the maximum constraint violation v(x) as:

varphi(x) = f(x)                        if v(x) <= v1
varphi(x) = f(x) + 1e5 * (v(x) - v1)   if v1 < v(x) <= v2
varphi(x) = np.inf                       if v(x) > v2

where v1 = min(0.01, 1e-10 * max(1, v0)), v2 = max(0.1, 2 * v0), and v0 is the maximum constraint violation at the initial guess.

n_jobsint, optional

The number of parallel jobs to run the test. Default is the default number of workers in the default local cluster.

normalized_scoresbool, optional

Whether to normalize the scores of the solvers by the maximum score of the solvers. Default is True.

project_x0bool, optional

Whether to project the initial point to the feasible set. Default is False.

run_plainbool, optional

Whether to run an extra experiment with the ‘plain’ feature. Default is False.

savepathstr, optional

The path to store the results. Default is the current working directory.

score_funcallable, optional

The scoring function to calculate the scores of the solvers. It should be a callable profile_scores -> solver_scores, where profile_scores is a 4D array containing scores for all profiles. The first dimension of profile_scores corresponds to the index of the solver, the second corresponds to the index of tolerance starting from 1, the third represents history-based or output-based profiles, and the fourth represents performance profiles, data profiles, or log-ratio profiles. The default scoring function takes the average of the history-based performance profiles under all the tolerances.

score_onlybool, optional

Whether to only calculate the scores of the solvers without drawing the profiles and saving the data. Default is False.

score_weight_funcallable, optional

The weight function to calculate the scores of the solvers in the performance and data profiles. It should be a callable representing a nonnegative function in R^+. Default is a constant function returning 1.

seedint, optional

The seed of the random number generator. Default is 0.

semilogxbool, optional

Whether to use the semilogx scale during plotting profiles (performance profiles and data profiles). Default is True.

silentbool, optional

Whether to show the information of the progress. Default is False.

solver_israndlist of bool, optional

Whether the solvers are randomized or not. Default is a list of bools of the same length as the number of solvers, where the value is True if the solver is randomized, and False otherwise. Note that if ‘n_runs’ is not specified, we will set it 5 for the randomized solvers.

solver_nameslist of str, optional

The names of the solvers. Default is the names of the callables in solvers.

solver_verboseint, optional

The level of the verbosity of the solvers. 0 means no verbosity, 1 means some verbosity, and 2 means full verbosity. Default is 1.

solvers_to_loadlist of int, optional

The indices of the solvers to load when the ‘load’ option is provided. It can be a list of different integers selected from 0 to the total number of solvers minus 1 of the loading experiment. At least two indices should be provided. Default is all the solvers.

summarize_data_profilesbool, optional

Whether to add all the data profiles to the summary PDF. Default is True.

summarize_log_ratio_profilesbool, optional

Whether to add all the log-ratio profiles to the summary PDF. Default is False.

summarize_output_based_profilesbool, optional

Whether to add all the output-based profiles of the selected profiles to the summary PDF. Default is True.

summarize_performance_profilesbool, optional

Whether to add all the performance profiles to the summary PDF. Default is True.

xlabel_data_profilestr, optional

The label of the x-axis of the data profiles. Default is ‘Number of simplex gradients’. Note: LaTeX formatting is supported. The same applies to the options ‘xlabel_log_ratio_profile’, ‘xlabel_performance_profile’, ‘ylabel_data_profile’, ‘ylabel_log_ratio_profile’, and ‘ylabel_performance_profile’.

xlabel_log_ratio_profilestr, optional

The label of the x-axis of the log-ratio profiles. Default is ‘Problem’.

xlabel_performance_profilestr, optional

The label of the x-axis of the performance profiles. Default is ‘Performance ratio’.

ylabel_data_profilestr, optional

The label of the y-axis of the data profiles. Default is ‘Data profiles ($mathrm{tol} = %s$)’, where ‘%s’ will be replaced by the current tolerance in LaTeX format. You can also use ‘%s’ in your custom label, and it will be replaced accordingly. The same applies to the options ‘ylabel_log_ratio_profile’ and ‘ylabel_performance_profile’.

ylabel_log_ratio_profilestr, optional

The label of the y-axis of the log-ratio profiles. Default is ‘Log-ratio profiles ($mathrm{tol} = %s$)’, where ‘%s’ will be replaced by the current tolerance in LaTeX format.

ylabel_performance_profilestr, optional

The label of the y-axis of the performance profiles. Default is ‘Performance profiles ($mathrm{tol} = %s$)’, where ‘%s’ will be replaced by the current tolerance in LaTeX format.

*Options for problems:*

Options in this part are used to select problems for benchmarking.

First select which problem libraries to use based on the ‘plibs’

option. Then select problems from these libraries according to the

given options (‘problem_names’, ‘ptype’, ‘mindim’, ‘maxdim’, ‘minb’,

‘maxb’, ‘minlcon’, ‘maxlcon’, ‘minnlcon’, ‘maxnlcon’, ‘mincon’,

‘maxcon’, and ‘excludelist’).

plibslist of str, optional

The problem libraries to be used. It should be a list of strs. The built-in choices are 's2mpj', 'pycutest', and 'custom'. Default setting is 's2mpj'. Note that 'pycutest' requires the separate installation of the pycutest package; see https://jfowkes.github.io/pycutest/ for installation instructions. You can also use your own problem library by specifying its name here together with the custom_problem_libs_path option.

ptypestr, optional

The type of the problems to be selected. It should be a str consisting of any combination of ‘u’ (unconstrained), ‘b’ (bound constrained), ‘l’ (linearly constrained), and ‘n’ (nonlinearly constrained), such as ‘b’, ‘ul’, ‘ubn’. Default is ‘u’.

mindimint, optional

The minimum dimension of the problems to be selected. Default is 1.

maxdimint, optional

The maximum dimension of the problems to be selected. Default is mindim + 1.

minbint, optional

The minimum number of bound constraints of the problems to be selected. Default is 0.

maxbint, optional

The maximum number of bound constraints of the problems to be selected. Default is minb + 10.

minlconint, optional

The minimum number of linear constraints of the problems to be selected. Default is 0.

maxlconint, optional

The maximum number of linear constraints of the problems to be selected. Default is minlcon + 10.

minnlconint, optional

The minimum number of nonlinear constraints of the problems to be selected. Default is 0.

maxnlconint, optional

The maximum number of nonlinear constraints of the problems to be selected. Default is minnlcon + 10.

minconint, optional

The minimum number of linear and nonlinear constraints of the problems to be selected. Default is min(minlcon, minnlcon).

maxconint, optional

The maximum number of linear and nonlinear constraints of the problems to be selected. Default is max(maxlcon, maxnlcon).

custom_problem_libs_pathstr or Path, optional

The path to a directory containing custom problem libraries. Each subdirectory in this path should be a problem library with the same structure as the built-in libraries (e.g., ‘s2mpj’, ‘pycutest’, ‘custom’). Specifically, each subdirectory should contain a file named ‘<library_name>_tools.py’ with two functions: ‘<library_name>_load’ and ‘<library_name>_select’. This option allows users to use their own problem libraries without modifying the installed package. Default is None, meaning only built-in libraries are available.

excludelistlist, optional

The list of problems to be excluded. Default is not to exclude any problem.

problem_nameslist of str, optional

The names of the problems to be selected. It should be a list of strs. Default is not to select any problem by name but by the options above.

problemProblem, optional

A problem to be benchmarked. It should be an instance of the class Problem. If it is provided, we will only run the test on this problem with the given feature and draw the history plots. Default is not to set any problem.

Raises:

TypeError

If an argument received an invalid value.

ValueError

If the arguments are inconsistent.

See also

Problem

Representation of optimization problems.

Feature

Feature applied to problems during benchmarking.

FeaturedProblem

Problem equipped with a specific feature.

Notes

The current version supports benchmarking derivative-free optimization solvers.

Caution

The log-ratio profiles are available only when there are exactly two solvers. For more information on performance and data profiles, see [1], [2], [5]. For that of log-ratio profiles, see [4], [6].

Caution

All callable arguments (solvers, distribution, mod_x0, mod_affine, mod_bounds, mod_linear_ub, mod_linear_eq, mod_fun, mod_cub, mod_ceq, merit_fun, score_fun, score_weight_fun) must be picklable for parallel execution (n_jobs > 1). In particular, lambda functions are not picklable and will cause the benchmark to fall back to sequential mode automatically. To take advantage of parallel execution, define named functions (using def) instead of lambda expressions.

1. Two problem libraries are available by default: S2MPJ (see [3]) and PyCUTEst (Linux and macOS only). To use your own problem library, see the custom_problem_libs_path option or the guide on our website.

2. Each problem library has a config.txt file that controls options such as variable_size and test_feasibility_problems. You can override these at runtime using set_plib_config or by setting the corresponding environment variables (e.g., S2MPJ_VARIABLE_SIZE). See get_plib_config and set_plib_config for details.

3. When the load option is provided, the function loads data from a previous experiment and draws profiles using the provided options. Available options in this mode are:

   • Profile and plot options: benchmark_id, solver_names, feature_stamp, errorbar_type, savepath, max_tol_order, merit_fun, run_plain, score_only, summarize_performance_profiles, summarize_data_profiles, summarize_log_ratio_profiles, summarize_output_based_profiles, silent, semilogx, normalized_scores, score_weight_fun, score_fun, solvers_to_load, line_colors, line_styles, line_widths, bar_colors.

   • Feature options: none.

   • Problem options: plibs, ptype, mindim, maxdim, minb, maxb, minlcon, maxlcon, minnlcon, maxnlcon, mincon, maxcon, excludelist.

4. More information about OptiProfiler can be found at https://www.optprof.com.

References

[1] (1,2)

E. D. Dolan and J. J. Moré. Benchmarking optimization software with performance profiles. Math. Program., 91(2):201–213, 2002. doi:10.1007/s101070100263 <https://doi.org/10.1007/s101070100263>.

[2] (1,2)

N. Gould and J. Scott. A note on performance profiles for benchmarking software. ACM Trans. Math. Software, 43(2):15:1–5, 2016. doi:10.1145/2950048 <https://doi.org/10.1145/2950048>.

[3]

S. Gratton and Ph. L. Toint. S2MPJ and CUTEst optimization problems for Matlab, Python and Julia. arXiv:2407.07812, 2024.

[4] (1,2)

J. L. Morales. A numerical study of limited memory BFGS methods. Appl. Math. Lett., 15(4):481–487, 2002. doi:10.1016/S0893-9659(01)00162-8 <https://doi.org/10.1016/S0893-9659(01)00162-8>.

[5] (1,2)

J. J. Moré and S. M. Wild. Benchmarking derivative-free optimization algorithms. SIAM J. Optim., 20(1):172–191, 2009. doi:10.1137/080724083 <https://doi.org/10.1137/080724083>.

[6]

H.-J. M. Shi, M. Q. Xuan, F. Oztoprak, and J. Nocedal. On the numerical performance of finite-difference-based methods for derivative-free optimization. Optim. Methods Softw., 38(2):289–311, 2023. doi:10.1080/10556788.2022.2121832 <https://doi.org/10.1080/10556788.2022.2121832>.

Examples

Benchmark two toy solvers that randomly sample around the initial point. Save the following as a .py file and run it (the if __name__ == '__main__' guard is required on macOS/Windows because benchmark uses multiprocessing):

import numpy as np
from optiprofiler import benchmark

def solver1(fun, x0):
    rng = np.random.default_rng(0)
    best_x, best_f = x0, fun(x0)
    for _ in range(10 * len(x0)):
        x = x0 + rng.standard_normal(len(x0))
        if fun(x) < best_f:
            best_x, best_f = x, fun(x)
    return best_x

def solver2(fun, x0):
    rng = np.random.default_rng(0)
    best_x, best_f = x0, fun(x0)
    for _ in range(20 * len(x0)):
        x = x0 + rng.standard_normal(len(x0))
        if fun(x) < best_f:
            best_x, best_f = x, fun(x)
    return best_x

if __name__ == '__main__':
    scores = benchmark([solver1, solver2])