Contributing to Nevergrad
We want to make contributing to this project as easy and transparent as possible.
Whether you want to contribute or not, don’t hesitate to join Nevergrad users’ Facebook group
Our Development Process
nevergrad in development mode (if you wish to contribute to it), clone the repository and run
pip install -e .[all] from inside the repository folder,
pip install -e '.[all]' if you use
zsh. If the install fails because of
torch, you can preinstall it with the instructions on Pytorch website
Most of the code is covered by unit tests. You can run them with:
pytest nevergrad --cov=nevergrad
You can then run
mypy --implicit-reexport nevergrad
If you are not familiar with type checking, we do not want it to be an annoyance and you can can ignore errors by adding
# type: ignore at the end of lines flagged as incorrect.
If we consider it useful to have correct typing, we will update the code after your pull request is merged.
If you are however familiar with type hints, you can check with the strict mode:
mypy --implicit-reexport --strict nevergrad, but at any given moment all the code is not guaranteed to pass the test (we try to work on it regularly, see #409).
Unit tests and type checks (in non-strict mode) will be automatically run every time a pull request is submitted/updated.
Finally, we use pre-commit hooks to make sure the code follows the same coding style. We currently use black and
pylint. To install them, just run
pre-commit install once, and they will be activated for all your commits on this repository.
black compliance is automatically checked on all PRs. If you do not use precommit hooks, you
pip install black make your code compliant by running:
We actively welcome your pull requests.
Fork the repo and create your branch from
If you’ve added code that should be tested, add tests.
If you’ve changed APIs, update the documentation.
Ensure the test suite passes.
Make sure your code lints.
If you haven’t already, complete the Contributor License Agreement (“CLA”).
Contributor License Agreement (“CLA”)
In order to accept your pull request, we need you to submit a CLA. You only need to do this once to work on any of Facebook’s open source projects.
Complete your CLA here: <https://code.facebook.com/cla>
We use GitHub issues to track public bugs. Please ensure your description is clear and has sufficient instructions to be able to reproduce the issue.
Facebook has a bounty program for the safe disclosure of security bugs. In those cases, please go through the process outlined on that page and do not file a public issue.
We use pep8, but allow lines to be as long as 140 characters. Please use the pre-commit hooks to ensure correctness (see section “Our Development Process”).
Documentation can be build with
make html from the
By contributing to
nevergrad, you agree that your contributions will be licensed
under the LICENSE file in the root directory of this source tree.
Adding an algorithm
The following guidelines are for people who want to add an algorithm to
nevergrad. They may be already outdated, feel free to update them if you find them unclear or think they should evolve.
Where to add the algorithm?
All optimizers are implemented in the
ng.optimization subpackage, and all optimizer classes are available either in the
ng.optimization.optimizerlib module (which is aliased to
ng.optimizers, or through the optimizer registry:
Implementations are however spread into several files:
optimizerlib.py: this is the default file, where most algorithms are implemented. It also imports optimizers from all other files.
oneshot.py: this is where one-shot optimizers are implemented
differentialevolution.py: this is where differential evolution algorithms are implemented.
recastlib.py: this is where we implement ask & tell versions of existing Python implementations which do not follow this pattern. The underlying class which helps spawn a subprocess to run the existing implementation into is in recaster.py. Hopefully, you won’t need this.
If you implement one new algorithm and if this algorithm is not one-shot/evolutionary/recast, you should implement it into optimizerlib.py. If you implement a whole family of algorithms, you are welcome to create a new corresponding file. Still, this structure is not final, it is bound to evolve and you are welcome to amend it.
How to implement it?
Base class features
All algorithms derive from a base class named
Optimizer and are registered through a decorator. The implementation of the base class is here.
This base class implements the
It records a sample of the best evaluated points through the
archive attribute of class
Archive. The archive can be seen be used as if it was of type
Dict[np.ndarray, Value], but since
np.ndarray are not hashable, the underlying implementation converts arrays into bytes and
register them into the
Archive however does not implement
because converting from bytes to array is not very efficient, one should therefore integrate on
bytesdict and the keys can then be
transformed back to arrays using
OnePlusOne implementation for an example.
The key tuple if the point location, and
Value is a class with attributes:
count: number of evaluations at this point.
mean: mean value of the evaluations at this point.
variance: variance of the evaluations at this point.
For more details, see the implementation in utils.py.
Through the archive, you can therefore access most useful information about past evaluations. A pruning mechanism makes sure this archive does
not grow too much. This pruning can be tuned through the
pruning attribute of the optimizer.
By default it keeps at least the best 100 points and cleans up when reaching 1000 points. It can be straightforwardly deactivated by setting optimizer’s archive
Optimizer class also tracks the best optimistic and pessimistic points through the
current_bests attribute which is of type:
Dict[str, Point]. The key string is either
pessimistic, and the
Point value is a
Value with an additional
x attribute, recording the location of the point.
Methods and attributes
4 methods are designed to be overridden:
__init__: for the initialization of your algorithm
_internal_ask_candidate: to fetch the next point to be evaluated. This function is the only one that is absolutely required to be overridden. The default
askmethod calls this method (please do not override the default
_internal_tell_candidate: to update your algorithm with the new point. The default
tellmethod calls this internal method after updating the archive (see paragraph above), please do not override it.
_internal_provide_recommendation: to provide the final recommendation. By default, the recommendation is the pessimistic best point.
_internal_tell_not_asked(optional): if the optimizer must handle points differently if they were not asked for, this method must be implemented. If you do not want to support this, you can raise
base.TellNotAskedNotSupportedError. A unit test will make sure that the optimizer either accepts the point or raises this error.
These functions work with
Parameter instances, which hold the parameter(s)
value (which can also be accessed through
kwargs) depending on the parametrization provided at the initialization of the optimizer.
New instances of
Parameter can be easily created through the
optimizer.parametrization.spawn_child(). This way it keeps track of the
filiation between parameters. The value can then be updated either directly through the
parameter.value attribute, or by setting
the value in the “standardized space” (parameter.set_standardized_data).
If the algorithm is not able to handle parallelization (if
ask cannot be called multiple times consecutively), the
no_parallelization class attribute must be set to
Seeding has an important part for the significance and reproducibility of the algorithm benchmarking. We want to ensure the following constraints:
we expect stochastic algorithms to be actually stochastic, if we set a hard seed inside the implementation this assumption is broken.
we need the randomness to obtain relevant statistics when benchmarking the algorithms on deterministic functions.
we should be able to seed from outside when we need it: we expect that setting a seed to the global random state should lead to reproducible results.
In order to facilitate these behaviors, each parametrization has a
random_state attribute (np.random.RandomState), which can be seeded by the
user if need be.
optimizer._rng is a shortcut to access it. All calls to stochastic functions should there be made through it.
By default, it will be seeded randomly by drawing a number from the global numpy random state so
that seeding the global numpy random state will yield reproducible results as well
A unit tests automatically makes sure that all optimizers have repeatable behaviors on a simple test case when seeded from outside (see below).
About type hints
We have used type hints throughout
nevergrad to make it more robust, and the continuous integration will check that everything is correct when pull requests are submitted.
If you need to add base types for your code, please import them through
import nevergrad.common.typing as tp.
However, we do not want typing to be an annoyance for contributors who do not care about it, so please feel entirely free to use
# type: ignore on each line the continuous integration will flag as incorrect, so that the errors disappear. If we consider it useful to have correct typing, we will update the code after your pull request is merged.
If it makes sense to create several variations of your optimizer, using different hyperparameters, you can implement an
OptimizerFamily. The only aim of this class is to create
Optimizers and set the parameters before returning it. This is still an experimental API which may evolve soon, and an example can be found in the implementation of differential evolution algorithms.
How to test it
You are welcome to add tests if you want to make sure your implementation is correct. It is however not required since some tests are run on all registered algorithms. They will test two features:
that all algorithms are able to find the optimum of a simple 2-variable quadratic fitness function.
that running the algorithms twice after setting a seed lead to the exact same recommendation. This is useful to make sure we will get repeatability in the benchmarks.
To run these tests, you can use:
The repeatability test will however crash the first time you run it, since no value for the recommendation of your algorithm exists. This is automatically added when running the tests, and if everything goes well the second time you run them, it means everything is fine. You will see in you diff that an additional line was added to a file containing all expected recommendations.
If for any reason one of this test is not suitable for your algorithm, we’ll discuss this in the pull request and decide of the appropriate workaround.
How to benchmark it
Benchmarks are implemented in two files experiments.py and frozenexperiments.py. While the former can be freely modified (benchmarks will be regularly added and removed), the latter file implements experiments which should not be modified when adding an algorithm, because they are used in tests, or for reproducibility of published results.
Providing some benchmark results along your pull requests will highlight the interest of your algorithm. It is however not required. For now, there is no standard approach for benchmarking your algorithm. You can implement your own benchmark, or copy an existing one and add your algorithm. Feel free to propose other solutions.
How benchmarks are implemented
A benchmark is made of many
Experiment instances. An
Experiment is basically the combination of a test function, and settings for the optimization (optimizer, budget, etc…).
Benchmarks are specified using a generator of
Experiment instances. See examples in experiments.py. If you want to make sure your benchmark is perfectly reproducible, you will need to be careful of properly seeding the functions and/or the experiments.