Running Experiments with COCO¶

COCO provides an interface for running experiments. This interface fgeneric has been ported in different languages:

in Python, Matlab/GNU Octave, C/C++, R and Java,

exampleexperiment and exampletiming
Testing New Functions
- Testing Functions with Parameter

`exampleexperiment` and `exampletiming`¶

In each language, two example scripts are provided. Below are the example scripts in Matlab/GNU Octave:

exampleexperiment runs an experiment on one testbed,

% runs an entire experiment for benchmarking MY_OPTIMIZER
% on the noise-free testbed. fgeneric.m and benchmarks.m
% must be in the path of Matlab/Octave
% CAPITALIZATION indicates code adaptations to be made

addpath('PUT_PATH_TO_BBOB/matlab');  % should point to fgeneric.m etc.
datapath = 'PUT_MY_BBOB_DATA_PATH';  % different folder for each experiment
% opt.inputFormat = 'row';
opt.algName = 'PUT ALGORITHM NAME';
opt.comments = 'PUT MORE DETAILED INFORMATION, PARAMETER SETTINGS ETC';
maxfunevals = '10 * dim'; % 10*dim is a short test-experiment taking a few minutes 
                          % INCREMENT maxfunevals successively to larger value(s)
minfunevals = 'dim + 2';  % PUT MINIMAL SENSIBLE NUMBER OF EVALUATIONS for a restart
maxrestarts = 1e4;        % SET to zero for an entirely deterministic algorithm

dimensions = [2, 3, 5, 10, 20, 40];  % small dimensions first, for CPU reasons
functions = benchmarks('FunctionIndices');  % or benchmarksnoisy(...)
instances = [1:5, 41:50];  % 15 function instances

more off;  % in octave pagination is on by default

t0 = clock;
rand('state', sum(100 * t0));

for dim = dimensions
  for ifun = functions
    for iinstance = instances
      fgeneric('initialize', ifun, iinstance, datapath, opt); 

      % independent restarts until maxfunevals or ftarget is reached
      for restarts = 0:maxrestarts
        if restarts > 0  % write additional restarted info
          fgeneric('restart', 'independent restart')
        end
        MY_OPTIMIZER('fgeneric', dim, fgeneric('ftarget'), ...
                     eval(maxfunevals) - fgeneric('evaluations'));
        if fgeneric('fbest') < fgeneric('ftarget') || ...
           fgeneric('evaluations') + eval(minfunevals) > eval(maxfunevals)
          break;
        end  
      end

      disp(sprintf(['  f%d in %d-D, instance %d: FEs=%d with %d restarts,' ...
                    ' fbest-ftarget=%.4e, elapsed time [h]: %.2f'], ...
                   ifun, dim, iinstance, ...
                   fgeneric('evaluations'), ...
                   restarts, ...
                   fgeneric('fbest') - fgeneric('ftarget'), ...
                   etime(clock, t0)/60/60));

      fgeneric('finalize');
    end
    disp(['      date and time: ' num2str(clock, ' %.0f')]);
  end
  disp(sprintf('---- dimension %d-D done ----', dim));
end

exampletiming runs the CPU-timing experiment.

% runs the timing experiment for MY_OPTIMIZER. fgeneric.m
% and benchmarks.m must be in the path of MATLAB/Octave

addpath('PUT_PATH_TO_BBOB/matlab');  % should point to fgeneric.m etc.

more off;  % in octave pagination is on by default

timings = [];
runs = [];
dims = [];
for dim = [2,3,5,10,20,40]
  nbrun = 0;
  ftarget = fgeneric('initialize', 8, 1, 'tmp');
  tic;
  while toc < 30  % at least 30 seconds
    MY_OPTIMIZER(@fgeneric, dim, ftarget, 1e5);  % adjust maxfunevals
    nbrun = nbrun + 1;
  end  % while
  timings(end+1) = toc / fgeneric('evaluations');
  dims(end+1) = dim;    % not really needed
  runs(end+1) = nbrun;  % not really needed
  fgeneric('finalize');
  disp([['Dimensions:' sprintf(' %11d ', dims)]; ...
        ['      runs:' sprintf(' %11d ', runs)]; ...
        [' times [s]:' sprintf(' %11.1e ', timings)]]);
end

Matlab/GNU Octave ¶

The above example scripts run a complete experiment. The entire interface for running experiments is defined via the function fgeneric.m. fgeneric is initialized with a function number, instance number and output data path. fgeneric can then be called to evaluate the test function. The end of a run or trial is signaled by calling fgeneric with the ‘finalize’ keyword.

C/C++¶

The interface for running experiments relies on functions such as fgeneric_initialize, fgeneric_finalize, fgeneric_ftarget. The evaluation function is fgeneric_evaluate for a single vector or fgeneric_evaluate_vector for an array of vectors as input.

A specific folder structure is needed for running an experiment. This folder structure can be obtained by un-tarring the archive createfolders.tar.gz and renaming the output folder or alternatively by executing the Python script createfolders.py before executing any experiment program. Make sure createfolders.py is in your current working directory and from the command-line simply execute:

$ python createfolders.py FOLDERNAME
FOLDERNAME was created.

The code provided can be compiled in C or C++.

R ¶

Quick Installation ¶

Run the following commands in your R session:

install.packages(c("BBmisc", "stringr"),
                 repos="http://cran.at.r-project.org")
fn <- file.path(tempdir(), "bbob_current.tar.gz")
download.file("http://coco.lri.fr/downloads/download15.03/bbobr.tar.gz",
              destfile=fn)
install.packages(fn, repos=NULL)
file.remove(fn)

You should now be able to load the package by running

library("bbob")

If all went well, you can skip down to the next section. Otherwise consulte the detailed instructions which follow.

Detailed Installation ¶

Before you start, install the required dependencies. At the R prompt enter

install.packages(c("BBmisc", "stringr"))

This should download and install the two packages. Now download the current BBOB R source package and save it somewhere on your hard drive. The most up-to-date version is always available here, but there should also be a stable version available on the COCO website. Now it’s time to install the package. In R, run the following command:

install.packages("/path/to/bbob_current.tar.gz", repos=NULL)

Note that you will have to adjust the path and possibly the filename to match the location where you stored the downloaded package on your hard drive. On Windows you will also need to have the Rtools installed for this to work since the package contains C code. If you have any problems with this step, do not hesitate to contact me for assistance.

After completing the above steps, you should be able to load the package in R

library("bbob")

The help page for the bbo_benchmark function should get you started if you do not want to continue reading this introduction.

?bbo_benchmark

Simple experiment ¶

If you already have an optimizer ready to go in R and all you want to do is produce a BBOB dataset for post-processing, then this section will walk you through the required steps. We will use the high-level interface bbo_benchmark in this example. If you want to parallelize your optimization runs, need to perform complex initializations or just want full control, skip down to the next section for a brief tour of the low-level interface.

In this example we will use the L-BFGS-B optimizer included in base R. You will need to adapt parts of the code to suit your optimizer. The first step is to wrap your optimization algorithm in a function with the following signature

function(par, fun, lower, upper, max_eval)

where par will be a numeric vector with the starting point for the optimization, fun is the function to be minimized, lower and upper are the bounds of the box constraints and max_eval is the number of function evaluations left in the allocated budget. What these five parameters mean is best conveyed by an example. Here we wrap the optim function included in base R

my_optimizer <- function(par, fun, lower, upper, max_eval) {
  optim(par, fun, method="L-BFGS-B",
        lower=lower, upper=upper, control=list(maxit=max_eval))
}

If your algorithm does not have the notion of an initial parameter setting, you can safely ignore the par parameter in your implementation. You might also notice, that we do not strictly adhere to the max_eval limit because the number of iterations is generally not equal to the number of function evaluations for L-BFGS-B. This is OK. max_eval is only a hint to the optimizer how much effort it should put into optimizing the function.

Should your algorithm perform restarts internally it is possible to log these using the bbob_log_restart function. The function takes exactly one argument, a string describing the reason for the restart.

We are now ready to run our experiment. This is done by calling the bbo_benchmark function

bbo_benchmark(my_optimizer, "l-bfgs-b", "optim_l-bfgs-b")

which will perform the BBOB experiments (caution, may take many hours). The first argument passed to bbo_benchmark is our optimization wrapper from the previous step. Make sure that it has the correct function signature! Next we supply the so called algorithm id. This is a short descriptive name for our optimization algorithm. Ideally it should include the package name and version which contains the algorithm. So for for genoud from the rgenoud package, we might use rgenoud::genoud (5.7-3) as the algorithm id. The last required argument is the name of the base directory where the result files will be stored. Again, it is a good idea to include the algorithm name in the directory name. If no other arguments are given, this runs a complete BBO benchmark on the noiseless test functions. This includes all instances (1-5 and 21-30), all dimensions (2, 3, 5, 10, 20, 40). If you do not want to include runs in 40 dimensions or want to use different instances you can change the defaults using the dimensions and instances arguments to bbo_benchmark. For details, see the manual page for bbo_benchmark.

If no other budget is specified, bbo_benchmark will perform random independent restarts of your algorithm until the desired target precision (1e-8) is reached or the default budget of 10000000 function evaluations is exhausted. If you want to reduce the budget, you can by specifiying it as the budget argument to bbo_benchmark.

To run the required timing experiment, execute the following code:

bbo_timing(my_optimizer)

It will return a data frame with the relevant timing information.

Low-level interface ¶

Will follow soon.

Python ¶

The interface for running an experiment is fgeneric which is used within exampleexperiment.py:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

"""Runs an entire experiment for benchmarking PURE_RANDOM_SEARCH on a testbed.

CAPITALIZATION indicates code adaptations to be made.
This script as well as files bbobbenchmarks.py and fgeneric.py need to be
in the current working directory.

Under unix-like systems: 
    nohup nice python exampleexperiment.py [data_path [dimensions [functions [instances]]]] > output.txt &

"""
import sys # in case we want to control what to run via command line args
import time
import numpy as np
import fgeneric
import bbobbenchmarks

argv = sys.argv[1:] # shortcut for input arguments

datapath = 'PUT_MY_BBOB_DATA_PATH' if len(argv) < 1 else argv[0]

dimensions = (2, 3, 5, 10, 20, 40) if len(argv) < 2 else eval(argv[1])
function_ids = bbobbenchmarks.nfreeIDs if len(argv) < 3 else eval(argv[2])  
# function_ids = bbobbenchmarks.noisyIDs if len(argv) < 3 else eval(argv[2])
instances = range(1, 6) + range(41, 51) if len(argv) < 4 else eval(argv[3])

opts = dict(algid='PUT ALGORITHM NAME',
            comments='PUT MORE DETAILED INFORMATION, PARAMETER SETTINGS ETC')
maxfunevals = '10 * dim' # 10*dim is a short test-experiment taking a few minutes 
# INCREMENT maxfunevals SUCCESSIVELY to larger value(s)
minfunevals = 'dim + 2'  # PUT MINIMAL sensible number of EVALUATIONS before to restart
maxrestarts = 10000      # SET to zero if algorithm is entirely deterministic 


def run_optimizer(fun, dim, maxfunevals, ftarget=-np.Inf):
    """start the optimizer, allowing for some preparation. 
    This implementation is an empty template to be filled 
    
    """
    # prepare
    x_start = 8. * np.random.rand(dim) - 4
    
    # call, REPLACE with optimizer to be tested
    PURE_RANDOM_SEARCH(fun, x_start, maxfunevals, ftarget)

def PURE_RANDOM_SEARCH(fun, x, maxfunevals, ftarget):
    """samples new points uniformly randomly in [-5,5]^dim and evaluates
    them on fun until maxfunevals or ftarget is reached, or until
    1e8 * dim function evaluations are conducted.

    """
    dim = len(x)
    maxfunevals = min(1e8 * dim, maxfunevals)
    popsize = min(maxfunevals, 200)
    fbest = np.inf
    
    for _ in range(0, int(np.ceil(maxfunevals / popsize))):
        xpop = 10. * np.random.rand(popsize, dim) - 5.
        fvalues = fun(xpop)
        idx = np.argsort(fvalues)
        if fbest > fvalues[idx[0]]:
            fbest = fvalues[idx[0]]
            xbest = xpop[idx[0]]
        if fbest < ftarget:  # task achieved 
            break

    return xbest

t0 = time.time()
np.random.seed(int(t0))

f = fgeneric.LoggingFunction(datapath, **opts)
for dim in dimensions:  # small dimensions first, for CPU reasons
    for fun_id in function_ids:
        for iinstance in instances:
            f.setfun(*bbobbenchmarks.instantiate(fun_id, iinstance=iinstance))

            # independent restarts until maxfunevals or ftarget is reached
            for restarts in xrange(maxrestarts + 1):
                if restarts > 0:
                    f.restart('independent restart')  # additional info
                run_optimizer(f.evalfun, dim,  eval(maxfunevals) - f.evaluations,
                              f.ftarget)
                if (f.fbest < f.ftarget
                    or f.evaluations + eval(minfunevals) > eval(maxfunevals)):
                    break

            f.finalizerun()

            print('  f%d in %d-D, instance %d: FEs=%d with %d restarts, '
                  'fbest-ftarget=%.4e, elapsed time [h]: %.2f'
                  % (fun_id, dim, iinstance, f.evaluations, restarts,
                     f.fbest - f.ftarget, (time.time()-t0)/60./60.))

        print '      date and time: %s' % (time.asctime())
    print '---- dimension %d-D done ----' % dim

Testing New Functions ¶

We describe here how to use fgeneric to record experiments on functions that are not part of the BBOB testbeds.

Note

This feature is only available in Python for the moment.

Example: log experiment using the Nelder-Mead simplex algorithm (scipy.optimize.fmin) on the sphere function. The following commands from the Python Interpreter does 15 runs of the Nelder-Mead simplex algorithm on the 2-D sphere functions. The data is recorded in folder data in the current working directory.

>>> from pylab import *
>>> import fgeneric as fg
>>> import scipy.optimize as so
>>> f = lambda x: sum(i**2 for i in x) # function definition
>>> e = fg.LoggingFunction(datapath='data', algid='Nelder-Mead simplex',
                  comments='x0 uniformly sampled in [0, 1]^2, '
                           'default settings')
>>> for i in range(15): # 15 repetitions
...     e.setfun(fun=f, fopt=0., funId='sphere', iinstance='0')
...     so.fmin(e.evalfun, x0=rand(2)) # algorithm call
...     e.finalizerun()
(<bound method LoggingFunction.evalfun of <fgeneric.LoggingFunction object at [...]>>, 1e-08)
Optimization terminated successfully.
         Current function value: 0.000000
         Iterations: [...]
         Function evaluations: [...]
array([...])
[...]
>>> # Display convergence graphs
>>> import bbob_pproc as bb
>>> ds = bb.load('data')
>>> ds.plot()

(Source code, png, hires.png, pdf)

Testing Functions with Parameter ¶

Note

This feature is only available in Python for the moment.

Example: log experiment using the BFGS algorithm (scipy.optimize.fmin) on the ellipsoid function with different condition numbers.

The following commands from the Python Interpreter does 15 runs of the BFGS algorithm on the 2-D sphere functions. The data is recorded in folder data in the current working directory and generate

>>> from pylab import *
>>> import fgeneric as fg
>>> import scipy.optimize as so
>>> import numpy as np

>>> e = fg.LoggingFunction(datapath='ellipsoid', algid='BFGS',
                  comments='x0 uniformly sampled in [0, 1]^5, default settings')
>>> cond_num = 10**np.arange(0, 7)
>>> for c in cond_num:
...     f = lambda x: np.sum(c**np.linspace(0, 1, len(x)) * x**2)
...     # function definition: these are term-by-term operations
...     for i in range(5): # 5 repetitions
...         e.setfun(fun=f, fopt=0., funId='ellipsoid', iinstance=0,
...                  condnum=c)
...         so.fmin_bfgs(e.evalfun, x0=np.random.rand(5)) # algorithm call
...         e.finalizerun()
(<bound method LoggingFunction.evalfun of <fgeneric.LoggingFunction object at [...]>>, 1e-08)
Optimization terminated successfully.
         Current function value: 0.000000
         Iterations: [...]
         Function evaluations: [...]
         Gradient evaluations: [...]
array([...])
[...]
>>> # plot data
>>> import bbob_pproc as bb
>>> import bbob_pproc.ppfigparam
>>> ds = bb.load('ellipsoid')
>>> bb.ppfigparam.plot(ds, param='condnum')
>>> bb.ppfigparam.beautify()
>>> import matplotlib.pyplot as plt
>>> plt.xlabel('Condition Number')

(Source code, png, hires.png, pdf)

Running Experiments with COCO¶

`exampleexperiment` and `exampletiming`¶

Matlab/GNU Octave ¶

C/C++¶

R ¶

Quick Installation ¶

Detailed Installation ¶

Simple experiment ¶

Low-level interface ¶

Python ¶

Testing New Functions ¶

Testing Functions with Parameter ¶

Table Of Contents

Previous topic

Next topic

This Page