Factorial ANOVA
(including two-way ANOVA)

1 Continuous Dependent Variable with normal distribution

2
or more Categorical Independent Variables

Factorial ANOVA measures whether a combination of
independent variables predict the value of a dependent variable. The term “way”
is often used to describe the number of independent variables measured by an
ANOVA test. For example, one-way ANOVA measures the effect of one independent
variable on a dependent variable, and two-way ANOVA measures the effect of two
independent variables on the dependent variable. In practice, ANOVA with
greater than three independent variables is rarely used because of the
complexity of interpreting the results. [9]

HCI Example:

In ‘This Computer Responds to User Frustration’
[5], Klein et al test whether computer software,
programmed to apologize after it performs poorly, affects subjects’ subsequent
use of the software. Subjects were told they would test a new computer game.
While playing, half of the subjects experienced a game where their character
froze for short periods of time, but the game clock continued to run,
presumably causing user frustration.

After playing the game for several minutes,
subjects completed an online post-game questionnaire in one of three
randomly-selected formats. The first survey format consisted of closed-ended
questions asking the user about his/her experience with the game. The second
survey format included open-ended questions about the game, providing an
opportunity for the subject to “vent” his/her frustration with the game if
he/she so chose. Finally, the third survey format generated an apology if the
subject responded negatively to a question.
To illustrate, a subject might have received the response, “the computer
apologizes to you for its part in giving you a crummy experience” after the
subject gave a negative rating.

Upon completion of the survey, subjects were
allowed to continue playing the game for as long as they wished. Post-survey
game play time served as the experiment’s dependent variable; the hypothesis
was subjects would play the game longer after the post-survey if they received
an apology after giving a negative response in the post-game questionnaire.

The research team used factorial ANOVA to prove
the hypothesis. The game-delay/no-game delay condition served as one of the
independent variables, and the post-game survey format served as the other,
creating 6 possible game variations:

NO DELAY in game 1,
plus DELAY
in game 1, plus

Closed-ended survey questions,
or Closed-ended survey
questions, or

Open-ended survey questions, or Open-ended survey questions, or

Survey reacts with empathy Survey reacts with empathy

The factorial ANOVA model represents a 2 x 3
design because one independent variable, game delay, has 2 levels, and the
other, post-survey format, has 3. If, for example, on the other hand, an experiment
had 3 independent variables, each with 2 levels, the design would be
characterized as 2 x 2 x 2.

The experiment results are as follows:

From
Klein et al. [5]

The graphic supports several conclusions that
typify results from a factorial ANOVA experiment:

·
If subjects experienced no delays in game 1,
then compared to other subjects, they played a shorter game 2

·
The survey format condition affected the amount
of game 2 play time, regardless of the game 1 ‘delay’ condition

·
The ANOVA results are significant, but there is
not a linear trend from left to right with respect to the ‘survey format’
condition. This is because ANOVA only tests whether individual condition means
vary from the aggregate sample mean. A separate post-test must be conducted if
the experimenter wants to test whether results from ANOVA follow a significant
linear trend.

In summary, the experiment controlled two independent
variables, game delay and post-game questionnaire format. The research team
used factorial ANOVA to show how the second game play time was affected by the
independent variables. A normal distribution can be assumed when sample size is
greater than 30, and this study included 70 subjects. For all of these reasons,
factorial ANOVA proves to be a valid statistic for measuring the experiment’s
results.

Values to report when using factorial ANOVA:

·
F value, with degrees of freedom (e.g. F (x, y)
= z, where x and y are the degrees of freedom)

·
p value