Factorial ANOVA (including two-way ANOVA)

           1 Continuous Dependent Variable with normal distribution

                       2 or more Categorical Independent Variables

 

Factorial ANOVA measures whether a combination of independent variables predict the value of a dependent variable. The term “way” is often used to describe the number of independent variables measured by an ANOVA test. For example, one-way ANOVA measures the effect of one independent variable on a dependent variable, and two-way ANOVA measures the effect of two independent variables on the dependent variable. In practice, ANOVA with greater than three independent variables is rarely used because of the complexity of interpreting the results. [9]

 

HCI Example:

In ‘This Computer Responds to User Frustration’ [5], Klein et al test whether computer software, programmed to apologize after it performs poorly, affects subjects’ subsequent use of the software. Subjects were told they would test a new computer game. While playing, half of the subjects experienced a game where their character froze for short periods of time, but the game clock continued to run, presumably causing user frustration.

 

After playing the game for several minutes, subjects completed an online post-game questionnaire in one of three randomly-selected formats. The first survey format consisted of closed-ended questions asking the user about his/her experience with the game. The second survey format included open-ended questions about the game, providing an opportunity for the subject to “vent” his/her frustration with the game if he/she so chose. Finally, the third survey format generated an apology if the subject responded negatively to a question.  To illustrate, a subject might have received the response, “the computer apologizes to you for its part in giving you a crummy experience” after the subject gave a negative rating.

 

Upon completion of the survey, subjects were allowed to continue playing the game for as long as they wished. Post-survey game play time served as the experiment’s dependent variable; the hypothesis was subjects would play the game longer after the post-survey if they received an apology after giving a negative response in the post-game questionnaire.

 

The research team used factorial ANOVA to prove the hypothesis. The game-delay/no-game delay condition served as one of the independent variables, and the post-game survey format served as the other, creating 6 possible game variations:

 

NO DELAY in game 1, plus                                DELAY in game 1, plus

    Closed-ended  survey questions, or                 Closed-ended  survey questions, or

    Open-ended survey questions, or                    Open-ended survey questions, or

    Survey reacts with empathy                               Survey reacts with empathy

 

The factorial ANOVA model represents a 2 x 3 design because one independent variable, game delay, has 2 levels, and the other, post-survey format, has 3. If, for example, on the other hand, an experiment had 3 independent variables, each with 2 levels, the design would be characterized as 2 x 2 x 2.

 

 

 

The experiment results are as follows:

 

From Klein et al. [5]

 

The graphic supports several conclusions that typify results from a factorial ANOVA experiment:

 

·         If subjects experienced no delays in game 1, then compared to other subjects, they played a shorter game 2

·         The survey format condition affected the amount of game 2 play time, regardless of the game 1 ‘delay’ condition

·         The ANOVA results are significant, but there is not a linear trend from left to right with respect to the ‘survey format’ condition. This is because ANOVA only tests whether individual condition means vary from the aggregate sample mean. A separate post-test must be conducted if the experimenter wants to test whether results from ANOVA follow a significant linear trend.

 

In summary, the experiment controlled two independent variables, game delay and post-game questionnaire format. The research team used factorial ANOVA to show how the second game play time was affected by the independent variables. A normal distribution can be assumed when sample size is greater than 30, and this study included 70 subjects. For all of these reasons, factorial ANOVA proves to be a valid statistic for measuring the experiment’s results.

 

 

 

Values to report when using factorial ANOVA:

 

·         F value, with degrees of freedom (e.g. F (x, y) = z, where x and y are the degrees of freedom)

·          p value