# Running Experiments with COCO¶

COCO provides an interface for running experiments. This interface `fgeneric`

has been
ported in different languages:

- in Python, Matlab/GNU Octave, C/C++, R and Java,

`exampleexperiment`

and `exampletiming`

¶

In each language, two example scripts are provided. Below are the example scripts in Matlab/GNU Octave:

`exampleexperiment`

runs an experiment on one testbed,

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 | ```
% runs an entire experiment for benchmarking MY_OPTIMIZER
% on the noise-free testbed. fgeneric.m and benchmarks.m
% must be in the path of Matlab/Octave
% CAPITALIZATION indicates code adaptations to be made
addpath('PUT_PATH_TO_BBOB/matlab'); % should point to fgeneric.m etc.
datapath = 'PUT_MY_BBOB_DATA_PATH'; % different folder for each experiment
% opt.inputFormat = 'row';
opt.algName = 'PUT ALGORITHM NAME';
opt.comments = 'PUT MORE DETAILED INFORMATION, PARAMETER SETTINGS ETC';
maxfunevals = '10 * dim'; % 10*dim is a short test-experiment taking a few minutes
% INCREMENT maxfunevals successively to larger value(s)
minfunevals = 'dim + 2'; % PUT MINIMAL SENSIBLE NUMBER OF EVALUATIONS for a restart
maxrestarts = 1e4; % SET to zero for an entirely deterministic algorithm
dimensions = [2, 3, 5, 10, 20, 40]; % small dimensions first, for CPU reasons
functions = benchmarks('FunctionIndices'); % or benchmarksnoisy(...)
instances = [1:5, 41:50]; % 15 function instances
more off; % in octave pagination is on by default
t0 = clock;
rand('state', sum(100 * t0));
for dim = dimensions
for ifun = functions
for iinstance = instances
fgeneric('initialize', ifun, iinstance, datapath, opt);
% independent restarts until maxfunevals or ftarget is reached
for restarts = 0:maxrestarts
if restarts > 0 % write additional restarted info
fgeneric('restart', 'independent restart')
end
MY_OPTIMIZER('fgeneric', dim, fgeneric('ftarget'), ...
eval(maxfunevals) - fgeneric('evaluations'));
if fgeneric('fbest') < fgeneric('ftarget') || ...
fgeneric('evaluations') + eval(minfunevals) > eval(maxfunevals)
break;
end
end
disp(sprintf([' f%d in %d-D, instance %d: FEs=%d with %d restarts,' ...
' fbest-ftarget=%.4e, elapsed time [h]: %.2f'], ...
ifun, dim, iinstance, ...
fgeneric('evaluations'), ...
restarts, ...
fgeneric('fbest') - fgeneric('ftarget'), ...
etime(clock, t0)/60/60));
fgeneric('finalize');
end
disp([' date and time: ' num2str(clock, ' %.0f')]);
end
disp(sprintf('---- dimension %d-D done ----', dim));
end
``` |

`exampletiming`

runs the CPU-timing experiment.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | ```
% runs the timing experiment for MY_OPTIMIZER. fgeneric.m
% and benchmarks.m must be in the path of MATLAB/Octave
addpath('PUT_PATH_TO_BBOB/matlab'); % should point to fgeneric.m etc.
more off; % in octave pagination is on by default
timings = [];
runs = [];
dims = [];
for dim = [2,3,5,10,20,40]
nbrun = 0;
ftarget = fgeneric('initialize', 8, 1, 'tmp');
tic;
while toc < 30 % at least 30 seconds
MY_OPTIMIZER(@fgeneric, dim, ftarget, 1e5); % adjust maxfunevals
nbrun = nbrun + 1;
end % while
timings(end+1) = toc / fgeneric('evaluations');
dims(end+1) = dim; % not really needed
runs(end+1) = nbrun; % not really needed
fgeneric('finalize');
disp([['Dimensions:' sprintf(' %11d ', dims)]; ...
[' runs:' sprintf(' %11d ', runs)]; ...
[' times [s]:' sprintf(' %11.1e ', timings)]]);
end
``` |

### Matlab/GNU Octave¶

The above example scripts run a complete experiment. The entire interface
for running experiments is defined via the function `fgeneric.m`

.
`fgeneric`

is *initialized* with a function number, instance number and
output data path. `fgeneric`

can then be called to evaluate the test function.
The end of a run or trial is signaled by calling `fgeneric`

with the ‘finalize’
keyword.

### C/C++¶

The interface for running experiments relies on functions such as
`fgeneric_initialize`

, `fgeneric_finalize`

, `fgeneric_ftarget`

.
The evaluation function is `fgeneric_evaluate`

for a
single vector or `fgeneric_evaluate_vector`

for an array of vectors as input.

A specific folder structure is needed for running an experiment. This folder
structure can be obtained by un-tarring the archive `createfolders.tar.gz`

and renaming the output folder or alternatively by executing the Python script
`createfolders.py`

before executing any experiment program. Make sure
`createfolders.py`

is in your current working directory and from the
command-line simply execute:

```
$ python createfolders.py FOLDERNAME
FOLDERNAME was created.
```

The code provided can be compiled in C or C++.

### R¶

#### Quick Installation¶

Run the following commands in your R session:

```
install.packages(c("BBmisc", "stringr"),
repos="http://cran.at.r-project.org")
fn <- file.path(tempdir(), "bbob_current.tar.gz")
download.file("http://coco.lri.fr/downloads/download15.03/bbobr.tar.gz",
destfile=fn)
install.packages(fn, repos=NULL)
file.remove(fn)
```

You should now be able to load the package by running

```
library("bbob")
```

If all went well, you can skip down to the next section. Otherwise consulte the detailed instructions which follow.

#### Detailed Installation¶

Before you start, install the required dependencies. At the R prompt enter

```
install.packages(c("BBmisc", "stringr"))
```

This should download and install the two packages. Now download the current BBOB R source package and save it somewhere on your hard drive. The most up-to-date version is always available here, but there should also be a stable version available on the COCO website. Now it’s time to install the package. In R, run the following command:

```
install.packages("/path/to/bbob_current.tar.gz", repos=NULL)
```

Note that you will have to adjust the path and possibly the filename to match the location where you stored the downloaded package on your hard drive. On Windows you will also need to have the Rtools installed for this to work since the package contains C code. If you have any problems with this step, do not hesitate to contact me for assistance.

After completing the above steps, you should be able to load the package in R

```
library("bbob")
```

The help page for the bbo_benchmark function should get you started if you do not want to continue reading this introduction.

```
?bbo_benchmark
```

#### Simple experiment¶

If you already have an optimizer ready to go in R and all you want to do is produce a BBOB dataset for post-processing, then this section will walk you through the required steps. We will use the high-level interface bbo_benchmark in this example. If you want to parallelize your optimization runs, need to perform complex initializations or just want full control, skip down to the next section for a brief tour of the low-level interface.

In this example we will use the L-BFGS-B optimizer included in base R. You will need to adapt parts of the code to suit your optimizer. The first step is to wrap your optimization algorithm in a function with the following signature

```
function(par, fun, lower, upper, max_eval)
```

where par will be a numeric vector with the starting point for the optimization, fun is the function to be minimized, lower and upper are the bounds of the box constraints and max_eval is the number of function evaluations left in the allocated budget. What these five parameters mean is best conveyed by an example. Here we wrap the optim function included in base R

```
my_optimizer <- function(par, fun, lower, upper, max_eval) {
optim(par, fun, method="L-BFGS-B",
lower=lower, upper=upper, control=list(maxit=max_eval))
}
```

If your algorithm does not have the notion of an initial parameter setting, you can safely ignore the par parameter in your implementation. You might also notice, that we do not strictly adhere to the max_eval limit because the number of iterations is generally not equal to the number of function evaluations for L-BFGS-B. This is OK. max_eval is only a hint to the optimizer how much effort it should put into optimizing the function.

Should your algorithm perform restarts internally it is possible to log these using the bbob_log_restart function. The function takes exactly one argument, a string describing the reason for the restart.

We are now ready to run our experiment. This is done by calling the bbo_benchmark function

```
bbo_benchmark(my_optimizer, "l-bfgs-b", "optim_l-bfgs-b")
```

which will perform the BBOB experiments (caution, may take many
hours). The first argument passed to bbo_benchmark is our
optimization wrapper from the previous step. Make sure that it has the
correct function signature! Next we supply the so called *algorithm
id*. This is a short descriptive name for our optimization
algorithm. Ideally it should include the package name and version
which contains the algorithm. So for for genoud from the rgenoud
package, we might use rgenoud::genoud (5.7-3) as the algorithm
id. The last required argument is the name of the base directory where
the result files will be stored. Again, it is a good idea to include
the algorithm name in the directory name. If no other arguments are
given, this runs a complete BBO benchmark on the noiseless test
functions. This includes all instances (1-5 and 21-30), all dimensions
(2, 3, 5, 10, 20, 40). If you do not want to include runs in 40
dimensions or want to use different instances you can change the
defaults using the dimensions and instances arguments to
bbo_benchmark. For details, see the manual page for bbo_benchmark.

If no other budget is specified, bbo_benchmark will perform random independent restarts of your algorithm until the desired target precision (1e-8) is reached or the default budget of 10000000 function evaluations is exhausted. If you want to reduce the budget, you can by specifiying it as the budget argument to bbo_benchmark.

To run the required timing experiment, execute the following code:

```
bbo_timing(my_optimizer)
```

It will return a data frame with the relevant timing information.

#### Low-level interface¶

Will follow soon.

### Python¶

The interface for running an experiment is `fgeneric`

which is used
within `exampleexperiment.py`

:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 | ```
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""Runs an entire experiment for benchmarking PURE_RANDOM_SEARCH on a testbed.
CAPITALIZATION indicates code adaptations to be made.
This script as well as files bbobbenchmarks.py and fgeneric.py need to be
in the current working directory.
Under unix-like systems:
nohup nice python exampleexperiment.py [data_path [dimensions [functions [instances]]]] > output.txt &
"""
import sys # in case we want to control what to run via command line args
import time
import numpy as np
import fgeneric
import bbobbenchmarks
argv = sys.argv[1:] # shortcut for input arguments
datapath = 'PUT_MY_BBOB_DATA_PATH' if len(argv) < 1 else argv[0]
dimensions = (2, 3, 5, 10, 20, 40) if len(argv) < 2 else eval(argv[1])
function_ids = bbobbenchmarks.nfreeIDs if len(argv) < 3 else eval(argv[2])
# function_ids = bbobbenchmarks.noisyIDs if len(argv) < 3 else eval(argv[2])
instances = range(1, 6) + range(41, 51) if len(argv) < 4 else eval(argv[3])
opts = dict(algid='PUT ALGORITHM NAME',
comments='PUT MORE DETAILED INFORMATION, PARAMETER SETTINGS ETC')
maxfunevals = '10 * dim' # 10*dim is a short test-experiment taking a few minutes
# INCREMENT maxfunevals SUCCESSIVELY to larger value(s)
minfunevals = 'dim + 2' # PUT MINIMAL sensible number of EVALUATIONS before to restart
maxrestarts = 10000 # SET to zero if algorithm is entirely deterministic
def run_optimizer(fun, dim, maxfunevals, ftarget=-np.Inf):
"""start the optimizer, allowing for some preparation.
This implementation is an empty template to be filled
"""
# prepare
x_start = 8. * np.random.rand(dim) - 4
# call, REPLACE with optimizer to be tested
PURE_RANDOM_SEARCH(fun, x_start, maxfunevals, ftarget)
def PURE_RANDOM_SEARCH(fun, x, maxfunevals, ftarget):
"""samples new points uniformly randomly in [-5,5]^dim and evaluates
them on fun until maxfunevals or ftarget is reached, or until
1e8 * dim function evaluations are conducted.
"""
dim = len(x)
maxfunevals = min(1e8 * dim, maxfunevals)
popsize = min(maxfunevals, 200)
fbest = np.inf
for _ in range(0, int(np.ceil(maxfunevals / popsize))):
xpop = 10. * np.random.rand(popsize, dim) - 5.
fvalues = fun(xpop)
idx = np.argsort(fvalues)
if fbest > fvalues[idx[0]]:
fbest = fvalues[idx[0]]
xbest = xpop[idx[0]]
if fbest < ftarget: # task achieved
break
return xbest
t0 = time.time()
np.random.seed(int(t0))
f = fgeneric.LoggingFunction(datapath, **opts)
for dim in dimensions: # small dimensions first, for CPU reasons
for fun_id in function_ids:
for iinstance in instances:
f.setfun(*bbobbenchmarks.instantiate(fun_id, iinstance=iinstance))
# independent restarts until maxfunevals or ftarget is reached
for restarts in xrange(maxrestarts + 1):
if restarts > 0:
f.restart('independent restart') # additional info
run_optimizer(f.evalfun, dim, eval(maxfunevals) - f.evaluations,
f.ftarget)
if (f.fbest < f.ftarget
or f.evaluations + eval(minfunevals) > eval(maxfunevals)):
break
f.finalizerun()
print(' f%d in %d-D, instance %d: FEs=%d with %d restarts, '
'fbest-ftarget=%.4e, elapsed time [h]: %.2f'
% (fun_id, dim, iinstance, f.evaluations, restarts,
f.fbest - f.ftarget, (time.time()-t0)/60./60.))
print ' date and time: %s' % (time.asctime())
print '---- dimension %d-D done ----' % dim
``` |

## Testing New Functions¶

We describe here how to use `fgeneric`

to record experiments on functions
that are not part of the BBOB testbeds.

Note

This feature is only available in Python for the moment.

Example: log experiment using the Nelder-Mead simplex algorithm
(`scipy.optimize.fmin`

) on the sphere function.
The following commands from the Python Interpreter does 15 runs of the
Nelder-Mead simplex algorithm on the 2-D sphere functions. The data is recorded
in folder `data`

in the current working directory.

```
>>> from pylab import *
>>> import fgeneric as fg
>>> import scipy.optimize as so
>>> f = lambda x: sum(i**2 for i in x) # function definition
>>> e = fg.LoggingFunction(datapath='data', algid='Nelder-Mead simplex',
comments='x0 uniformly sampled in [0, 1]^2, '
'default settings')
>>> for i in range(15): # 15 repetitions
... e.setfun(fun=f, fopt=0., funId='sphere', iinstance='0')
... so.fmin(e.evalfun, x0=rand(2)) # algorithm call
... e.finalizerun()
(<bound method LoggingFunction.evalfun of <fgeneric.LoggingFunction object at [...]>>, 1e-08)
Optimization terminated successfully.
Current function value: 0.000000
Iterations: [...]
Function evaluations: [...]
array([...])
[...]
>>> # Display convergence graphs
>>> import bbob_pproc as bb
>>> ds = bb.load('data')
>>> ds.plot()
```

(Source code, png, hires.png, pdf)

### Testing Functions with Parameter¶

Note

This feature is only available in Python for the moment.

Example: log experiment using the BFGS algorithm
(`scipy.optimize.fmin`

) on the ellipsoid function with different
condition numbers.

The following commands from the Python Interpreter does 15 runs of the
BFGS algorithm on the 2-D sphere functions. The data is recorded
in folder `data`

in the current working directory and generate

```
>>> from pylab import *
>>> import fgeneric as fg
>>> import scipy.optimize as so
>>> import numpy as np
>>> e = fg.LoggingFunction(datapath='ellipsoid', algid='BFGS',
comments='x0 uniformly sampled in [0, 1]^5, default settings')
>>> cond_num = 10**np.arange(0, 7)
>>> for c in cond_num:
... f = lambda x: np.sum(c**np.linspace(0, 1, len(x)) * x**2)
... # function definition: these are term-by-term operations
... for i in range(5): # 5 repetitions
... e.setfun(fun=f, fopt=0., funId='ellipsoid', iinstance=0,
... condnum=c)
... so.fmin_bfgs(e.evalfun, x0=np.random.rand(5)) # algorithm call
... e.finalizerun()
(<bound method LoggingFunction.evalfun of <fgeneric.LoggingFunction object at [...]>>, 1e-08)
Optimization terminated successfully.
Current function value: 0.000000
Iterations: [...]
Function evaluations: [...]
Gradient evaluations: [...]
array([...])
[...]
>>> # plot data
>>> import bbob_pproc as bb
>>> import bbob_pproc.ppfigparam
>>> ds = bb.load('ellipsoid')
>>> bb.ppfigparam.plot(ds, param='condnum')
>>> bb.ppfigparam.beautify()
>>> import matplotlib.pyplot as plt
>>> plt.xlabel('Condition Number')
```

(Source code, png, hires.png, pdf)