optiprofiler.benchmark#
- optiprofiler.benchmark(solvers: list[Callable[[...], Any]] | None = None, /, **kwargs) tuple[ndarray, ndarray | None, list[dict] | None][source]#
Benchmark optimization solvers on a set of problems with specified features.
This function creates multiple profiles for benchmarking optimization solvers on a set of problems with different features. It generates performance profiles, data profiles, and log-ratio profiles [1], [2], [4], [5] for the given solvers on various test suites, returning solver scores based on the profiles.
- Parameters:
- solverslist of callable, optional if
loadis provided Solvers to benchmark. Each solver must be a callable accepting corresponding arguments depending on the test suite you choose:
for an unconstrained problem,
solver(fun, x0) -> numpy.ndarray, shape (n,), wherefunis the objective function accepting a 1-D array and returning a float, andx0is the initial guess (1-D array);for a bound-constrained problem,
solver(fun, x0, xl, xu) -> numpy.ndarray, shape (n,), wherexlandxuare the lower and upper bounds (1-D arrays, may contain-numpy.infornumpy.inf);for a linearly constrained problem,
solver(fun, x0, xl, xu, aub, bub, aeq, beq) -> numpy.ndarray, shape (n,), whereaubandaeqare the coefficient matrices of the linear inequality and equality constraints, andbubandbeqare the right-hand side vectors;for a nonlinearly constrained problem,
solver(fun, x0, xl, xu, aub, bub, aeq, beq, cub, ceq) -> numpy.ndarray, shape (n,), wherecubandceqare the nonlinear inequality and equality constraint functions accepting a 1-D array and returning a 1-D array.
All vectors and matrices mentioned above are
numpy.ndarray.If the
loadoption is provided, solvers can be None, in which case data from a previous experiment will be loaded to generate profiles.
- solverslist of callable, optional if
- Returns:
- solver_scoresnumpy.ndarray
Scores of the solvers based on the profiles. See ‘score_fun’ in ‘Other Parameters’ for more details.
- profile_scoresnumpy.ndarray or None
A 4D array containing scores for all profiles. The first dimension corresponds to the index of the solver, the second to the index of tolerance starting from 1, the third represents history-based or output-based profiles, and the fourth represents performance profiles, data profiles, or log-ratio profiles. A profile at a tolerance no solver meets scores 0.
- curveslist of dict or None
A list containing the curves of all the profiles. A profile at a tolerance no solver meets keeps None in place of its curves.
- Other Parameters:
- Feature options:
- feature_namestr, optional
Name of the feature to apply to problems. The available features are ‘plain’, ‘perturbed_x0’, ‘noisy’, ‘truncated’, ‘permuted’, ‘linearly_transformed’, ‘random_nan’, ‘unrelaxable_constraints’, ‘nonquantifiable_constraints’, ‘quantized’, and ‘custom’. Default is ‘plain’. Several features can be composed by joining their names with ‘+’, for example ‘noisy+truncated’: the first name is applied to the original problem first, so each objective value is made noisy and the noisy value is then truncated. Supplied feature options are passed to every stage that accepts them, ‘n_runs’ is global, and ‘plain’ stages are identities. See the user guide for the rules.
- featuredict, list, tuple or Feature, optional
Structured specification of the feature: one stage entry
{'name': ..., 'options': {...}}, an ordered list/tuple of entries (bare names allowed) or an already builtFeature(used as it is, without reparsing). Each stage owns the options inside its entry, so repeated stages and different values of a shared key are expressible;n_runsstays a top-level option.featureandfeature_namecannot both be given, flat stage options are rejected withfeature, andfeaturecannot be combined withload.- n_runsint, optional
The number of runs of the experiments with the given feature. This is an experiment option, never a feature option: the experiment plan records it once, no stage carries a run count (it is rejected inside a
featureentry and byFeature(..., n_runs=...)), and the plain reference ofrun_plainalways uses one run. Unless given, it is 5 whensolver_israndmarks a randomized solver (not in a load), and otherwise the established default of the feature: 5 for ‘perturbed_x0’, ‘noisy’ (1 whennoise_mode='deterministic'), ‘permuted’, ‘linearly_transformed’ (also whenrotated=False), ‘random_nan’ and ‘truncated’ withperturbed_trailing_digits=True; 1 for ‘plain’, ‘truncated’, ‘unrelaxable_constraints’, ‘nonquantifiable_constraints’, ‘quantized’ and ‘custom’ (although ‘custom’ counts as stochastic). A composition defaults to the largest default of its effective stages. A presentNoneis invalid.- distributionstr or callable, optional
The distribution of perturbation in ‘perturbed_x0’ feature or random noise in ‘noisy’ feature. It should be either a str (or char), or a callable
(random_stream, dimension) -> random vector, accepting a random_stream and the dimension of a problem and returning a random vector with the given dimension. In ‘perturbed_x0’ case, the str should be either ‘spherical’ or ‘gaussian’ (default is ‘spherical’). In ‘noisy’ case, the str should be either ‘gaussian’ or ‘uniform’ (default is ‘gaussian’), and the callable should accept a random stream and output size.- perturbation_levelfloat, optional
The magnitude of the perturbation to the initial guess in the ‘perturbed_x0’ feature. Default is 1e-3.
- noise_levelfloat, optional
The magnitude of the noise in the ‘noisy’ feature. Default is 1e-3.
- noise_typestr, optional
The type of the noise in the ‘noisy’ features. It should be either ‘absolute’, ‘relative’, or ‘mixed’. Default is ‘mixed’.
- noise_modestr, optional
The mode of the noise in the ‘noisy’ feature. It should be either ‘random’ or ‘deterministic’. Default is ‘random’. When it is ‘deterministic’ and n_runs is not specified, n_runs defaults to 1.
- noise_mapstr or callable, optional
The deterministic scalar noise map in the ‘noisy’ feature. It should be either ‘chebyshev’ or a callable
x -> noisereturning a real scalar. It is used only when noise_mode is ‘deterministic’. Default is ‘chebyshev’. The built-in ‘chebyshev’ map follows the deterministic noise model in Moré and Wild [5].- significant_digitsint, optional
The number of significant digits in the ‘truncated’ feature. Default is 6.
- perturbed_trailing_digitsbool, optional
Whether we will randomize the trailing digits of the objective function value in the ‘truncated’ feature. Default is False.
- rotatedbool, optional
Whether to use a random or given rotation matrix to rotate the coordinates of a problem in the ‘linearly_transformed’ feature. Default is True.
- condition_factorfloat, optional
The scaling factor of the condition number of the linear transformation in the ‘linearly_transformed’ feature. More specifically, the condition number of the linear transformation will be
2**sqrt(condition_factor * n / 2)for dimension n >= 2 (it is 1 when n = 1). Default is 0.- nan_ratefloat, optional
The probability that the evaluation of the objective function will return np.nan in the ‘random_nan’ feature. Default is 0.05.
- unrelaxable_boundsbool, optional
Whether the bound constraints are unrelaxable or not in the ‘unrelaxable_constraints’ feature. Default is True.
- unrelaxable_linear_constraintsbool, optional
Whether the linear constraints are unrelaxable or not in the ‘unrelaxable_constraints’ feature. Default is False.
- unrelaxable_nonlinear_constraintsbool, optional
Whether the nonlinear constraints are unrelaxable or not in the ‘unrelaxable_constraints’ feature. Default is False.
- mesh_sizefloat, optional
The size of the mesh in the ‘quantized’ feature. Default is 1e-3.
- mesh_typestr, optional
The type of the mesh in the ‘quantized’ feature. It should be either ‘absolute’ or ‘relative’. Default is ‘absolute’.
- ground_truthbool, optional
Whether the featured problem is the ground truth or not in the ‘quantized’ feature. Default is True.
- mod_x0callable, optional
The modifier function to modify the initial guess in the ‘custom’ feature. It should be a callable
(random_stream, problem) -> modified_x0, where problem is an instance of the class Problem, and modified_x0 is the modified initial guess. No default.- mod_affinecallable, optional
The modifier function to generate the affine transformation applied to the variables in the ‘custom’ feature. It should be a callable
(random_stream, problem) -> (A, b, inv), where problem is an instance of the class Problem, A is the matrix of the affine transformation, b is the vector of the affine transformation, and inv is a candidate inverse of A. No default. The callable is asked once per problem and seed, and the initial point, the bounds, the linear constraints and every evaluation use that one answer. The triple is validated when the problem is built, and an error is raised unless the arrays are real, finite and of matching sizes,norm(abs(inv) @ abs(A), inf) < 1 / eps, and both inverse residuals pass the finite-precision checknorm(max(abs(A @ inv - I) - 64 * n * eps * abs(A) @ abs(inv), 0), 'fro') <= 1e-8 * nand the same forinv @ A. Hereepsis double-precision machine epsilon; these matrix-test quantities must be finite. Independently, rows then columns ofAare scaled by powers of two to formEwithout losing any entries, andcond(E, 1) < 1 / epsis required. These numerical checks are not a proof thatinvis an exact inverse. Integer and extended precision inputs are rounded to double precision before validation and use. IfAis exactly diagonal anddiag(inv)is its reciprocal to roundoff, the bounds stay bounds; otherwise every finite bound is posed as a linear constraint, so an off-diagonal entry ofAis never ignored, however small. Transport rejects finite bounds, coefficients or right-hand sides becoming non-finite, nonzero scaled bounds below the smallest normal magnitude (about 2.2e-308), and row or right-hand-side entries whose sum of absolute nonzero product terms lies below that magnitude. A strict original bound interval must not collapse to equal endpoints after translation or diagonal scaling. These are stage-local safeguards, not an exact transport certificate. Unlessmod_x0is supplied, a finite originalx0usesy = inv @ (x0 - b)unchanged only ify, its mapped point and the rounding allowance are finite, andabs(A @ y + b - x0)is at most both64 * n * eps * (abs(A) @ abs(y) + abs(b) + abs(x0))andsqrt(eps) * minimum(abs(x0), abs(x0 - b)), componentwise. There is no absolute floor: an error at a zero coordinate of eitherx0orx0 - btriggers a solve. Both scales are checked so a large coordinate origin cannot hide error in a small centered value. If either check fails,A @ y = x0 - bis solved independently and the solved point is checked for finiteness and the rounding allowance only; failure raises an initial-point error. The extra local threshold triggers a solve instead of trusting the inverse candidate; it is not a bound on the final forward error. Even an honestly computed inverse may trigger a solve, for example when a custom rotation mixes small and large coordinates. Intrinsic conditioning or mixed coordinate scales can cause forward drift even after solving.fun_initandmaxcv_initdescribe the posed start, not guaranteed recovery of an originally feasible point. Sincemod_linear_ubandmod_linear_eqreplace linear constraints, they are refused if they would remove generated bound rows, unlessmod_boundsalso takes ownership of the bounds. Derivatives follow the same map by the chain rule (seeFeaturedProblem). Construction checks do not recompute initial points or histories in previously saved experiments or built-object pickles; loading is not a migration of old results.- mod_boundscallable, optional
The modifier function to modify the bound constraints in the ‘custom’ feature. It should be a callable
(random_stream, problem) -> (modified_xl, modified_xu), where problem is an instance of the class Problem, modified_xl is the modified lower bound, and modified_xu is the modified upper bound. No default. A supplied modifier replaces its own component verbatim, andmod_boundsowns the logical original box in every representation. Whenmod_affineuses generated bound rows, the framework therefore does not add that original box again as linear rows; explicit linear constraints are still transported. The truth always scores the bounds of the problem, and the reference of the problem is unknown after anymod_bounds.- mod_linear_ubcallable, optional
The modifier function to modify the linear inequality constraints in the ‘custom’ feature. It should be a callable
(random_stream, problem) -> (modified_aub, modified_bub), where problem is an instance of the class Problem, modified_aub is the modified matrix of the linear inequality constraints, and modified_bub is the modified vector of the linear inequality constraints. No default.- mod_linear_eqcallable, optional
The modifier function to modify the linear equality constraints in the ‘custom’ feature. It should be a callable
(random_stream, problem) -> (modified_aeq, modified_beq), where problem is an instance of the class Problem, modified_aeq is the modified matrix of the linear equality constraints, and modified_beq is the modified vector of the linear equality constraints. No default.- mod_funcallable, optional
The modifier function to modify the objective function in the ‘custom’ feature. It should be a callable
(x, random_stream, problem) -> modified_fun, where x is the evaluation point, problem is an instance of the class Problem, and modified_fun is the modified objective function value. No default.- mod_cubcallable, optional
The modifier function to modify the nonlinear inequality constraints in the ‘custom’ feature. It should be a callable
(x, random_stream, problem) -> modified_cub, where x is the evaluation point, problem is an instance of the class Problem, and modified_cub is the modified vector of the nonlinear inequality constraints. No default.- mod_ceqcallable, optional
The modifier function to modify the nonlinear equality constraints in the ‘custom’ feature. It should be a callable
(x, random_stream, problem) -> modified_ceq, where x is the evaluation point, problem is an instance of the class Problem, and modified_ceq is the modified vector of the nonlinear equality constraints. No default.- Profile and plot options:
- bar_colorslist or numpy.ndarray, optional
Two different colors for the bars of two solvers in the log-ratio profiles. It can be a list of short names of colors (‘r’, ‘g’, ‘b’, ‘c’, ‘m’, ‘y’, ‘k’) or a 2-by-3 array with each row being a RGB triplet. Default is set to the ‘line_colors’ option. The bars cycle through the given colors, so with a single color both solvers’ bars use it.
- benchmark_idstr, optional
The identifier of the test. It is used to create the specific directory to store the results. Default is ‘out’ if the option ‘load’ is not provided, otherwise default is ‘.’.
- draw_hist_plotsstr, optional
Whether or how to draw the history plots of all the problems. It can be either ‘none’, ‘sequential’, or ‘parallel’. If it is ‘none’, we will not draw the history plots. If it is ‘parallel’, we will draw the history plots at the same time when solvers are solving the problems. If it is ‘sequential’, we will draw the history plots after all the problems are solved. For each problem, the original combined history plot is saved as
PROBLEM.pdf; raw-only and cumulative-minimum-only plots are saved under therawandcumminsubfolders. Default is ‘parallel’.- errorbar_typestr, optional
The type of the uncertainty interval that can be either ‘minmax’ or ‘meanstd’. When ‘n_runs’ is greater than 1, we run several times of the experiments and get average curves and uncertainty intervals. Default is ‘minmax’, meaning that we takes the pointwise minimum and maximum of the curves.
- feature_stampstr, optional
The stamp of the feature with the given options. It is used to create the specific directory to store the results. Default depends on features.
- hist_aggregationstr, optional
The aggregation method we use to reduce the number of points in the history plots. It can be ‘min’, ‘mean’, or ‘max’. Default is ‘min’.
- line_colorslist, optional
The colors of the lines in the plots. It can be a list of any valid matplotlib colors (short names, hex strings, RGB tuples, etc.). Default line colors are from the matplotlib tab10 color cycle. Note that if the number of solvers is greater than the number of colors, we will cycle through the colors.
- line_styleslist, optional
The Matplotlib line styles, such as ‘-’, ‘–’, ‘-.’, ‘:’, their names (e.g., ‘solid’ or ‘dashed’), or dash tuples (offset, dash_sequence). Color/marker format strings such as ‘r–’ or ‘-o’ are not line styles. Default line style order is [‘-’, ‘-.’, ‘–’, ‘:’]. Note that if the number of solvers is greater than the number of line styles, we will cycle through the styles.
- line_widthsfloat or list, optional
The widths of the lines in the plots. It should be a positive float or a list. Default is 1.5. Note that if the number of solvers is greater than the number of line widths, we will cycle through the widths.
- loadstr, optional
Loading the stored data from a completed experiment and draw profiles. It can be either ‘latest’ or a time stamp of an experiment in the format of ‘yyyyMMdd_HHmmss’, optionally followed by a collision suffix such as ‘_001’. No default. Note that if solvers is None, this key must be provided to load data from a previous experiment and generate profiles. The experiment is searched for in the current working directory joined with
benchmark_id(up to five levels of subdirectories), not insavepath. Load only archives from trusted sources: saved Python options may contain pickle objects, whose deserialization can execute code.- max_eval_factorint, optional
The factor multiplied to each problem’s dimension to get the maximum number of evaluations for each problem. Default is 500.
- max_tol_orderint, optional
The maximum order of the tolerance. In any profile (performance profiles, data profiles, and log-ratio profiles), we need to set a group of ‘tolerances’ to define the convergence test of the solvers. (Details can be found in the references.) We will set the tolerances as
10**(-k)fork = 1, 2, ..., max_tol_order. Default is 10.- merit_funcallable, optional
The merit function to measure the quality of a point using the objective function value and the maximum constraint violation. It should be a callable
(fun_value, maxcv_value, maxcv_init) -> merit_value, where fun_value is the objective function value, maxcv_value is the maximum constraint violation, and maxcv_init is the maximum constraint violation at the initial guess. The default merit function varphi(x) is defined by the objective function f(x) and the maximum constraint violation v(x) as:varphi(x) = f(x) if v(x) <= v1 varphi(x) = f(x) + 1e5 * (v(x) - v1) if v1 < v(x) <= v2 varphi(x) = np.inf if v(x) > v2
where v1 = min(0.01, 1e-10 * max(1, v0)), v2 = max(0.1, 2 * v0), and v0 is the maximum constraint violation at the initial guess. NaN objective or constraint values are invalid evaluations, including when a custom merit function returns a finite value for them. A run with an undefined initial value, including a merit function value of NaN at the initial guess, is not declared passing. Otherwise, if varphi(x_0) is +inf, the existing degenerate-test convention accepts valid evaluations; if it is -inf, only a merit value of -inf passes. Both cases are listed in
test_log/report.txt. In output-based profiles, an output obtained without any evaluation counts as one evaluation. A run whose solver raised an exception or returned nothing has no output: the initial point is recorded in its place, the run is listed intest_log/report.txtand it never passes the output-based test, whatever the target, while its history-based results stand. A solver that returns the initial point itself is not such a run. Withproblem, the score of a run whose initial merit is not finite is NaN (the relative decrease is undefined), whereas the profiles above apply the +inf convention.- n_jobsint, optional
The number of parallel jobs to run the test. Default is a conservative number of workers, chosen as about half of the available workers (at least 2 when more than one worker is available).
- normalized_scoresbool, optional
Whether to normalize the scores of the solvers by the maximum score of the solvers. Default is True.
- project_x0bool, optional
Whether to project the initial point to the feasible set. Default is False.
- run_plainbool, optional
Whether to run an extra experiment with the ‘plain’ feature. Default is False.
- savepathstr, optional
The path to store the results. Default is the current working directory. It does not affect where
loadsearches.- score_funcallable, optional
The scoring function to calculate the scores of the solvers. It should be a callable
profile_scores -> solver_scores, where profile_scores is a 4D array containing scores for all profiles. The first dimension of profile_scores corresponds to the index of the solver, the second corresponds to the index of tolerance starting from 1, the third represents history-based or output-based profiles, and the fourth represents performance profiles, data profiles, or log-ratio profiles. A history-based or output-based profile at a tolerance no solver meets has no curve and scores 0 for every solver. It must return one real score per solver: a vector of length n_solvers (a column or row array with n_solvers entries is flattened to that vector; any other shape raises an error). The default scoring function takes the average of the history-based performance profiles under all the tolerances.- score_onlybool, optional
Whether to only calculate the scores of the solvers without drawing the profiles and saving the data. Messages are printed to the console as in a saved run (subject to
silent), but no log file is written. Default is False.- plot_optionsdict, optional
Native Matplotlib property mappings for
figure,axes,xaxis,yaxis,xlabel,ylabel,title,text,lineandlegend; also acceptstick_paramsand a localrcmapping. For example,{'xlabel': {'fontsize': 18}, 'legend': {'fontsize': 14}}. Properties override package defaults for profiles, summaries and histories, including on load, without changing scores or source data. Legend entries remain solver-owned. Backend/interactive selection is not a style setting. Values must support serialization for parallel history rendering. Default is an empty mapping (existing style).- report_pathstr or pathlib.Path, optional
Write a compact, controller-private
eval_reportJSON and an adjacent<report-stem>.plot_data.jsonnumerical companion. Neither path may already exist. The main file holds exact scalar facts, existing scores, and references; the companion holds bounded history summaries and full prepared plot coordinates/bands. Verify its SHA256 and evaluation ID before following references. Reporting does not change returned values, scoring, seeds, or plotting options. Withscore_only=Trueonly these two explicitly requested JSON files are written; raw archives and PDFs are not enabled. Numerical, scoring, persistence and rendering outcomes remain separate. Neither file is automatically a public agent prompt. Default is no report.- score_weight_funcallable, optional
The weight function to calculate the scores of the solvers in the performance and data profiles. It should be a callable representing a nonnegative function in R^+. Default is a constant function returning 1.
- seedint, optional
The seed of the random number generator. Default is 0.
- semilogxbool, optional
Whether to use the semilogx scale during plotting profiles (performance profiles and data profiles). For data profiles, the plotted coordinate is
log2(1 + n_eval / (n + 1))but tick labels are displayed in the originaln_eval / (n + 1)units, typically0, 1, 2, 4, .... Default is True.- silentbool, optional
Whether to show the information of the progress. With True, no progress information is printed, also when loading saved results, and ‘solver_verbose’ is 0. Default is False.
- solver_israndlist of bool, optional
Whether the solvers are randomized or not. Default is a list of bools of the same length as the number of solvers, where the value is True if the solver is randomized, and False otherwise. Note that if ‘n_runs’ is not specified, we will set it 5 for the randomized solvers.
- solver_nameslist of str, optional
The names of the solvers. Default is the names of the callables in solvers.
- solver_verboseint, optional
The level of the verbosity of the solvers. 0 means no verbosity, 1 means some verbosity, and 2 means full verbosity. With ‘silent’ True the level is 0, so nothing of a solver is printed, and a level of 1 or 2 is replaced with a warning; a failing solver is still recorded in report.txt and the machine-readable report. Default is 1.
- solvers_to_loadlist of int, optional
The indices of the solvers to load when the ‘load’ option is provided. It can be a list of different integers selected from 0 to the total number of solvers minus 1 of the loading experiment. At least two indices should be provided. Default is all the solvers.
- summarize_data_profilesbool, optional
Whether to add all the data profiles to the summary PDF. Default is True.
- summarize_log_ratio_profilesbool, optional
Whether to add all the log-ratio profiles to the summary PDF. Default is False.
- summarize_output_based_profilesbool, optional
Whether to add all the output-based profiles of the selected profiles to the summary PDF. Default is True.
- summarize_performance_profilesbool, optional
Whether to add all the performance profiles to the summary PDF. Default is True.
- xlabel_data_profilestr, optional
The label of the x-axis of the data profiles. Default is ‘Number of simplex gradients’. Note: LaTeX formatting is supported. The same applies to the options ‘xlabel_log_ratio_profile’, ‘xlabel_performance_profile’, ‘ylabel_data_profile’, ‘ylabel_log_ratio_profile’, and ‘ylabel_performance_profile’. The text of the x-axis labels is passed to the renderer verbatim. Under LaTeX, write ‘%’ for a percent sign: a bare ‘%’ starts a LaTeX comment and cuts the label. Without LaTeX (see ‘text.usetex’ in the ‘rc’ group of ‘plot_options’), ‘%’ keeps its backslash; ‘$50%$’ works in both cases.
- xlabel_log_ratio_profilestr, optional
The label of the x-axis of the log-ratio profiles. Default is ‘Problem’.
- xlabel_performance_profilestr, optional
The label of the x-axis of the performance profiles. Default is ‘Performance ratio’.
- ylabel_data_profilestr, optional
The label of the y-axis of the data profiles. Default is ‘Data profiles ($mathrm{tol} = %s$)’, where ‘%s’ will be replaced by the current tolerance in LaTeX format. You can also use ‘%s’ in your custom label, and it will be replaced accordingly. Keep ‘%s’ inside ‘$…$’, as the defaults do, because the tolerance is LaTeX math: outside math mode it is literal text such as ‘10^{-1}’ without LaTeX, and it raises a LaTeX error under LaTeX. Only ‘%s’ is a placeholder: ‘%%’ and any other ‘%’ give a literal percent sign, escaped automatically where LaTeX or a ‘$…$’ segment needs it (so ‘50% solved at $%s$’ is a valid label). The same applies to the options ‘ylabel_log_ratio_profile’ and ‘ylabel_performance_profile’.
- ylabel_log_ratio_profilestr, optional
The label of the y-axis of the log-ratio profiles. Default is ‘Log-ratio profiles ($mathrm{tol} = %s$)’, where ‘%s’ will be replaced by the current tolerance in LaTeX format.
- ylabel_performance_profilestr, optional
The label of the y-axis of the performance profiles. Default is ‘Performance profiles ($mathrm{tol} = %s$)’, where ‘%s’ will be replaced by the current tolerance in LaTeX format.
- Problem options:
- Options in this part are used to select problems for benchmarking.
- First select which problem libraries to use based on the ‘plibs’
- option. Then select problems from these libraries according to the
- given options (‘problem_names’, ‘ptype’, ‘mindim’, ‘maxdim’, ‘minb’,
- ‘maxb’, ‘minlcon’, ‘maxlcon’, ‘minnlcon’, ‘maxnlcon’, ‘mincon’,
- ‘maxcon’, and ‘excludelist’). Library-specific selection and loading
- behavior is configured separately through ‘plib_options’.
- plibslist of str, optional
The problem libraries to be used. It should be a list of strs. Each name must be an ASCII Python identifier: it must start with a letter or underscore and contain only letters, digits, and underscores. The built-in choices are
's2mpj'and'custom'. Default setting is's2mpj'. Separately installed problem-library packages, including the OptiProfiler PyCUTEst, SOLAR, and experimental RS13 providers, are discovered through theoptiprofiler.problem_librariesentry-point group. You can also use an unpackaged local library by specifying its name here together with thecustom_problem_libs_pathoption.- plib_optionsmapping, optional
Per-experiment options grouped by problem-library name. Each value must be a mapping understood and validated by that library. For example, a hypothetical API-v1 plugin may accept
{'myproblems': {'variant': 'large'}}without changing another library or process-wide defaults. Keys for libraries not selected inplibsare rejected. Per-experiment values take precedence overset_plib_config(), environment-level settings, and package-owned defaults. The validated effective mapping, including defaults, is saved intest_log/options_refined.pkland in each library’s data intest_log/data_for_loading.h5. This option is rejected withproblemandload, because those modes do not invoke a problem library. Default is an empty mapping.- ptypestr, optional
The type of the problems to be selected. It should be a str consisting of any combination of ‘u’ (unconstrained), ‘b’ (bound constrained), ‘l’ (linearly constrained), and ‘n’ (nonlinearly constrained), such as ‘b’, ‘ul’, ‘ubn’. Default is ‘u’.
- mindimint, optional
The minimum dimension of the problems to be selected. It must be a finite integer greater than or equal to 1. Default is 1.
- maxdimint or positive infinity, optional
The maximum dimension of the problems to be selected. It must be an integer greater than or equal to 1, or positive infinity. A positive infinity (
np.inf) means that no maximum dimension is imposed. Default is mindim + 1.- minbint, optional
The minimum number of bound constraints of the problems to be selected. It must be a finite integer greater than or equal to 0. Default is 0.
- maxbint or positive infinity, optional
The maximum number of bound constraints of the problems to be selected. It must be an integer greater than or equal to 0, or positive infinity. A positive infinity (
np.inf) means that no maximum number of bound constraints is imposed. Default is minb + 10.- minlconint, optional
The minimum number of linear constraints of the problems to be selected. It must be a finite integer greater than or equal to 0. Default is 0.
- maxlconint or positive infinity, optional
The maximum number of linear constraints of the problems to be selected. It must be an integer greater than or equal to 0, or positive infinity. A positive infinity (
np.inf) means that no maximum number of linear constraints is imposed. Default is minlcon + 10.- minnlconint, optional
The minimum number of nonlinear constraints of the problems to be selected. It must be a finite integer greater than or equal to 0. Default is 0.
- maxnlconint or positive infinity, optional
The maximum number of nonlinear constraints of the problems to be selected. It must be an integer greater than or equal to 0, or positive infinity. A positive infinity (
np.inf) means that no maximum number of nonlinear constraints is imposed. Default is minnlcon + 10.- minconint, optional
The minimum number of linear and nonlinear constraints of the problems to be selected. It must be a finite integer greater than or equal to 0. Default is min(minlcon, minnlcon).
- maxconint or positive infinity, optional
The maximum number of linear and nonlinear constraints of the problems to be selected. It must be an integer greater than or equal to 0, or positive infinity. A positive infinity (
np.inf) means that no maximum number of constraints is imposed. Default is max(maxlcon, maxnlcon).- custom_problem_libs_pathstr or Path, optional
The path to a directory containing custom problem libraries, or a direct path to one custom problem library. Each custom problem library must contain a tools module named ‘<library_name>_tools.py’ with two functions: ‘<library_name>_load’ and ‘<library_name>_select’. This option allows users to use their own problem libraries without modifying the installed package. If an explicitly supplied custom library has the same name as another provider, the custom library is used for that run. Without an explicit custom path, duplicate providers are rejected as ambiguous, except that Python 3.8 and 3.9 cannot tell two distributions apart that declare the identical entry point, which therefore resolves. Default is None.
- excludelistlist, optional
The list of problems to be excluded. Default is not to exclude any problem.
- problem_nameslist of str, optional
The names of the problems to be selected. It should be a list of strs. Default is not to select any problem by name but by the options above.
- problemProblem, optional
A problem to be benchmarked. It should be an instance of the class Problem. If it is provided, we will only run the test on this problem with the given feature and draw the history plots. It cannot be combined with
load. Default is not to set any problem.
- Raises:
- TypeError
If an argument received an invalid value.
- ValueError
If the arguments are inconsistent.
See also
ProblemRepresentation of optimization problems.
FeatureFeature applied to problems during benchmarking.
FeaturedProblemProblem equipped with a specific feature.
Notes
The current version supports benchmarking derivative-free optimization solvers.
Caution
The log-ratio profiles are available only when there are exactly two solvers. For more information on performance and data profiles, see [1], [2], [5]. For that of log-ratio profiles, see [4], [6].
Caution
All callable arguments (
solvers,distribution,noise_map,mod_x0,mod_affine,mod_bounds,mod_linear_ub,mod_linear_eq,mod_fun,mod_cub,mod_ceq,merit_fun,score_fun,score_weight_fun) must be picklable for parallel execution (n_jobs > 1). In particular, lambda functions are not picklable and will cause the benchmark to fall back to sequential mode automatically. To take advantage of parallel execution, define named functions (usingdef) instead of lambda expressions.S2MPJ (see [3]) is the bundled default problem library. PyCUTEst, SOLAR (see [7]), and RS13 providers are distributed and tested independently; install a compatible provider package before selecting its public name. OptiProfiler discovers these packages through the versioned problem-library protocol. For local development without packaging, see the
custom_problem_libs_pathoption or the guide on our website. An explicitly supplied custom library overrides other providers with the same name for that run; all other duplicate providers are rejected, except that Python 3.8 and 3.9 cannot tell two distributions apart that declare the identical entry point, which therefore resolves.Problem libraries may define different library-specific options. For a reproducible run, provide them through
plib_options. Package-owned defaults may come fromconfig.txtor another plugin-specific source.set_plib_config()remains available for process-level defaults, and some adapters also support environment variables such asS2MPJ_VARIABLE_SIZE. The precedence isplib_optionsfirst, thenset_plib_config(), then the library-owned environment/configuration and defaults. Seeget_plib_config()for the current process-level values.When the
loadoption is provided, the function loads data from a previous experiment and draws profiles using the provided options. Available options in this mode are:Profile and plot options:
benchmark_id,solver_names,feature_stamp,errorbar_type,savepath,max_tol_order,merit_fun,run_plain,score_only,summarize_performance_profiles,summarize_data_profiles,summarize_log_ratio_profiles,summarize_output_based_profiles,silent,semilogx,normalized_scores,score_weight_fun,score_fun,solvers_to_load,line_colors,line_styles,line_widths,bar_colors.Feature options: none.
Problem options:
plibs,ptype,mindim,maxdim,minb,maxb,minlcon,maxlcon,minnlcon,maxnlcon,mincon,maxcon,excludelist.
More information about OptiProfiler can be found at https://www.optprof.com.
References
[1] (1,2)E. D. Dolan and J. J. Moré. Benchmarking optimization software with performance profiles. Math. Program., 91(2):201–213, 2002. doi:10.1007/s101070100263 <https://doi.org/10.1007/s101070100263>.
[2] (1,2)N. Gould and J. Scott. A note on performance profiles for benchmarking software. ACM Trans. Math. Software, 43(2):15:1–5, 2016. doi:10.1145/2950048 <https://doi.org/10.1145/2950048>.
[3]S. Gratton and Ph. L. Toint. S2MPJ and CUTEst optimization problems for Matlab, Python and Julia. arXiv:2407.07812, 2024.
[4] (1,2)J. L. Morales. A numerical study of limited memory BFGS methods. Appl. Math. Lett., 15(4):481–487, 2002. doi:10.1016/S0893-9659(01)00162-8 <https://doi.org/10.1016/S0893-9659(01)00162-8>.
[5] (1,2,3)J. J. Moré and S. M. Wild. Benchmarking derivative-free optimization algorithms. SIAM J. Optim., 20(1):172–191, 2009. doi:10.1137/080724083 <https://doi.org/10.1137/080724083>.
[6]H.-J. M. Shi, M. Q. Xuan, F. Oztoprak, and J. Nocedal. On the numerical performance of finite-difference-based methods for derivative-free optimization. Optim. Methods Softw., 38(2):289–311, 2023. doi:10.1080/10556788.2022.2121832 <https://doi.org/10.1080/10556788.2022.2121832>.
[7]N. Andrés-Thió, C. Audet, M. Diago, A. E. Gheribi, S. Le Digabel, X. Lebeuf, M. Lemyre-Garneau, and C. Tribes.
solar: A solar thermal power plant simulator for blackbox optimization benchmarking. Optimization and Engineering, 2025. doi:10.1007/s11081-024-09952-x <https://doi.org/10.1007/s11081-024-09952-x>.Examples
Benchmark two toy solvers that randomly sample around the initial point. Save the following as a
.pyfile and run it. Theif __name__ == '__main__'guard is required on every platform whenn_jobs > 1(the default on a multi-core machine): parallel workers are started with thespawnmethod, which imports the script again in each worker. A call reached while a worker imports the script raises aRuntimeErrorinstead of running the benchmark again.import numpy as np from optiprofiler import benchmark def solver1(fun, x0): rng = np.random.default_rng(0) best_x, best_f = x0, fun(x0) for _ in range(10 * len(x0)): x = x0 + rng.standard_normal(len(x0)) if fun(x) < best_f: best_x, best_f = x, fun(x) return best_x def solver2(fun, x0): rng = np.random.default_rng(0) best_x, best_f = x0, fun(x0) for _ in range(20 * len(x0)): x = x0 + rng.standard_normal(len(x0)) if fun(x) < best_f: best_x, best_f = x, fun(x) return best_x if __name__ == '__main__': scores = benchmark([solver1, solver2])