111 create yaml template for problem submission

[kvdblom]

Draft yaml template for review

[Dvermetten]

Maybe it would be useful to have some sort of comments to describe what the fields are / what would be valid input, specifically for the cases where there is a difference between quoted and non-quoted input. Example:

  variables: # information about the input variables
    types: continuous # can be one of (continuous, integer, binary, mixed)
    conditional: 'no' # whether there are conditional dependencies between variables, 'yes' or 'no'
    dimensionality: scalable # number of input variables, either as a number (in quotes) or scalable

[CIGbalance]

Thanks for this! The comments are mostly easy fixes or not that serious.

But I am mostly wondering of how this is supposed to be used? It is called a template, but it has BBOB-specific entries? I think this increases the risk of biasing the results.

I think I would suggest doing a template with a lot more comments (like @Dvermetten suggested) and then we can use this as real data as well as link to it as an example?

[CIGbalance]

I still think it would be best to enter problems separately, but I understand the necessity of streamlining this process. So if we want to allow entering suites, maybe we have different templates? Because some values might only make sense for a suite? Otherwise, for a suite, a lot of the input would be "varies" for number of objectives, variables, constraints, dynamic, noise, etc?

[kvdblom]

Yes, it would be nice to have problems separately, I have also prototyped how it might work to have both the suite and component problems, see here:

OPL/problems.yaml

Line 1107 in 84b34b6

problems:

I am not sure about separate templates for suites or problems, but we can think about this and discuss what makes sense. I would imagine, e.g., the BBOB sphere still has a bunch of "varies", because it is scalable in the number of variables for example. For a suite, I would want an exhaustive list (e.g., noise: yes, no / optional, or something like that) of the options, rather than a vague "varies". I will describe this in a comment in the template.

[CIGbalance]

This is confusing to enter, because they have boundary/box: yes.
I would suggest removing present, it should be clear from the rest of the values.

[kvdblom]

Yes, makes sense, we can also automatically derive the 'present' field from other fields later if we need it.

[kvdblom]

Actually, for backwards compatibility with the existing template it might make sense to keep it for the moment. What do you think @CIGbalance

[CIGbalance]

Technically, COCO has way more interfaces.
Also typo in languages.

[kvdblom]

Fixed typo.

Can you add those interfaces to the problems.yaml ?🙂

[kvdblom]

@CIGbalance

[CIGbalance]

This is going to be wild to analyse if we don't give some structure

[kvdblom]

What would you suggest? @CIGbalance

My initial idea would be to ask for something like:
Enter approximate time with units and greater/less than symbol if relevant, e.g.: <1s, or 2h5m3s

Are there good standards we can refer to/use?

[bn12]

Work in progress ...

`

Please enter the relevant information.

Fields that are not relevant can be left empty.

• name:
  short: BBOB
  full: Real-Parameter Black-Box Optimization Benchmarking
  suite/generator/single: suite
  objectives:
  number: '1'
  types: single
  1 - single
  2 - bi or multi or multiple
  3 - multi or multiple
  4 or more - many
  scalable - scalable
  variables:
  types: continuous
  continuous or discrete or mixed-integer
  conditional: 'no'
  yes or no
  dimensionality: scalable
  scalable or fixed number or interval or ranges (5-11,19,85)
  constraints:
  present: 'no'
  yes or no
  soft: '0'
  number of soft constraints
  hard: '0'
  number of hard constraints
  boundary/box: 'yes'
  yes or no, in case of yes the number should be equal to the dimensionality under variables
  permutation: 'no'
  yes or no, in case of yes the number should be euqal to the length of the permutation
  dynamic:
  present: 'no'
  yes or no
  types: ''
  ??
  noise: 'no'
  yes or no, guess we don't differentiate types of noise (by now?)
  modality:
  types: 'unimodal, multimodal'
  evaluations:
  multi-fidelity: 'no'
  partial possible: 'no'
  independent objectives: 'no'
  reference:
  links:
  - https://doi.org/10.1080/10556788.2020.1808977
  authors: ''
  contact person: ''
  implementations:
  • name: COCO
    link: https://github.com/numbbo/coco
    languges: 'C, Python'
    evaluation time: 'less than a second'
    specific requirements: 'no'
    source:
    real-world:
    degree: ''
    open/closed: ''
    artificial: 'yes'
    other: 'no'
    textual description:
    general info: ''
    motivation: 'evaluate algorithm performance for typical difficulties that occur in continuous problems'
    challenage/key characteristics: ''
    limitations: ''
    other info: ''
    `