The “experiments.yml” reference
Simexpal needs an experiments.yml
file to to automatically execute your experiments
on the desired instances. In this page we describe the structure of the experiments.yml
file. The experiments.yml
is a YAML file that contains a dictionary with several keys:
instances
: list of all the instances that will be used for the experimentsinstdir
: path to the directory that stores all the instancesexperiments
: list of all the experiments that will be executed on the instances
There are also further keys that allow for customization of the experiments and to automatically build the binaries the experiments run from:
builds
: list of Git builds, which also include build instructionsrevisions
: list of Git revisions of buildsvariants
: list of additional input parameters for experimentsmatrix
: specifies which combinations of experiments, instances, variants and revisions are run
Instances
This entry is a list of instances that will be used for experiments. The following keys are used for specifying instances:
extensions
: list of extensions that the instance hasfiles
: list of files the instance consists ofitems
: list of instancesname
: name of the instance (used when dealing with instances that consist of unrelated files)repo
: source of instancesset
: list of sets the instance belongs to
For detailed usage examples, see the Instances page.
Builds
This entry is a list of builds that will be used for revisions. The following keys are used for specifying builds:
configure
: list of dictionaries containing configuration parameterscompile
: list of dictionaries containing compilation parametersenviron
: dictionary of (environment variable, value)-pairsgit
: link to the Git repositoryinstall
: list of dictionaries containing installation parametersname
: (arbitrary) name of the buildrecursive-clone
: boolean (true
/false
) - whether to pull git submodules recursively or notregenerate
: list of dictionaries containing regeneration parametersrequires
: list of required buildsworkdir
: path of the working directory
For detailed usage examples, see the Builds page.
Revisions
This entry is a list of revisions that will be used for experiments. The following keys are used for specifying revisions:
build_version
: dictionary of (build, SHA-1 hash/branch)-pairsdevelop
: boolean (true
/false
) - whether this revision is a develop revision or notname
: (arbitrary) name of the revision
For detailed usage examples, see the Revisions page.
Experiments
This entry is a list of experiments that will be executed on all the instances. Each experiment includes three keys:
args
: list of experiment argumentsenviron
: dictionary of (environment variable, value)-pairsname
: name of the experimentnum_nodes
: number of nodes on which to runnum_threads
: number of cpus required per taskoutput
: dictionary containing all output file extensionsprocs_per_node
: number of tasks to invoke on each noderepeat
: integer - number of times an experiment is repeatedslurm_args
: list of additionalsbatch
argumentsstdout
: extension of the output filetimeout
: integer - timeout in secondsuse_builds
: list of used build names
For detailed usage examples, see the Experiments page.
Variants
This entry is a list of variants that will be used for experiments. The following keys are used for specifying variants:
axis
: name of the variant axisenum
: list of variant valuesenviron
: dictionary of (environment variable, value)-pairsextra_args
: list of variant argumentsitems
: list of dictionaries, which specify variants belonging to the same axis.name
: name of the variantnum_nodes
: number of nodes on which to runnum_threads
: number of cpus required per taskprocs_per_node
: number of tasks to invoke on each noderange
: list with start and stop elementsteps
: step size
For detailed usage examples, see the Variants page.
Run Matrix
This entry is a list of desired experiment combinations. The following keys are used for specifying desired experiment combinations:
axes
: list of included axis namesexperiments
: list of included experiment namesinclude
: list of dictionaries, which specify included experiment combinationsinstsets
: list of included instance set namesrepetitions
: integer - number of times all combinations of aninclude
entry are repeatedrevisions
: list of included revision namesvariants
: list of included variant names
For detailed usage examples, see the Run Matrix page.