Experiment design includes three different phases: define the goals of the experiment; identify and classify independent and dependent variables; choose an experiment design.

Icône de l'outil pédagogique Phase 1— Define the goals.
Of course, the selection of what conditions to run depends on the goals of the simulation. Why was the simulation model constructed? What particular issues are being examined during the current cycle of experimentation?  These goals generally belong to the upper level in the hierarchy of goals within the organization that is sponsoring the use of the simulation model. It is good to place these goals in perspective, to gain support for the effort that will be required, and to make sure that the short-term objectives are consistent with the overall goals of the organization (see Problem Definition process).

Icône de l'outil pédagogique Phase 2— Identify and classify variables.

The second step in the experiment design process is to identify quantities in the simulation that can be set to desired values (independent variables, e.g. input variables defining the policy options to be tested in SEAMLES-IF impact assessments) and the resulting system performance measures that are of interest (dependent variables), which are the selected (policy impact) indicators. Another class of variables to be considered when designing the experiment are variables which are known to affect the behaviour of the system. They are affected by the settings of the independent variables, but they are not considered dependent variables. They are often called “nuisance variables or outlook parameters”.

It is necessary to identify all variables of all three types before planning the set of runs. Dependent variables are determined by the objectives of the study. All independent variables should be identified, not just the ones that will be varied in the experiment. Nuisance variables must be monitored so that variation in the experiment results can be understood. In order to be able to reproduce the results later, the (fixed) values of any independent variables that were not adjusted must be recorded, as well as the values of ones that were varied. Independent and nuisance variables whose values are actually changed during the experiment will be called factors. Process diagrams and cause-effect diagrams can be used to identify them (see Barton, 1999).

Icône de l'outil pédagogique Phase 3— Choose an experiment design (matrix).

In this activity, one determines the number of distinct model settings to be run and the specific values of the factors for each of these runs. There are many strategies for selecting the number of runs and the factor settings for each run. Four types of experiment matrices are available, including the Hadamard matrix, Rechtschaffner matrix, fractional factorial matrix and the complete (full) factorial matrix.

In Table 1, an example of a fractional factorial design is presented. Factorial designs allow the modeller to analyse multi-level parameters influences and their dependencies, which is not possible in Hadamard and Rechtschaffner matrices. Factorial designs are based on a grid, with each factor tested in combination with every level of every other factor. Factorial designs are attractive for three reasons: 1) the number of levels that are required for each factor are one greater than the highest-order power of that variable in the model, and the resulting design permits the estimation of coefficients for all cross-product terms; 2) they are probably the most commonly used class of designs, and 3) the resulting set of run conditions are easy to visualize graphically for as many as nine factors.

Table 1 shows how resolution 3-factor fractional factorial designs can be constructed for a large number factor screening experiments (factor L = 6, resolution m = 3) where only main effects exist. The factorial design is based on the first 3 factors, A, B and C. The rest of the main effects (D, E and F) are assigned to the last three columns of AB, AC and BC respectively so their main effects are confounded with corresponding interaction effects. The level settings of main effects confounded with interaction effects at each design point are set to be the same as the level settings of the corresponding interaction effects. Thus, we have a design matrix and all main effects can be estimated.

Table 1. Resolution III Fractional Factorial Design.

The disadvantage of full factorial designs is that they require a large number of distinct runs. If a design includes all treatment combinations (with two levels per factor L), it is a full factorial design; if only a fraction of those is included, it is a fractional factorial design. Controlled sequential fractional-factorial designs (CSFD) have the flexibility to stop the experiment early, to increase the model size when further investigation is necessary, and to classify any desired main effect or interaction effect. For detailed description of factorial design and analysis, please refer to Montgomery (2001). Many books (e.g. Wu and Hamada, 2000) and software packages provide recommended designs for various resolutions and various values of L, the number of factors to be screened.

Régi par la licence Creative Commons Attribution Non-commercial Share Alike 3.0 License

copyright Seamless 2009