One-way ANOVA
1 Continuous Dependent Variable with
normal distribution
1
(Multi-level) Categorical Independent Variable
A one-way
analysis of variance (ANOVA) measures whether one or more components of a
multiple level independent variable predict the value of a dependent variable.
In
‘Comparative Effectiveness of Augmented Reality in Object Assembly’[12], Tang et al. test how long it takes subjects to
complete a 56 step assembly task using Duplo building blocks. Subjects receive
instructions for performing the task either via (1) a printed manual, (2)
computer assisted instruction (CAI) delivered on a traditional computer monitor
display, (3) CAI delivered on a head-mounted display, or (4) via Augmented
Reality (AR). In the AR condition, subjects use a head-mounted display that
overlays 3 dimensional instructions on the actual work space, including on the
Duplo blocks. The research team hypothesizes there will be a significant
difference between the amount of time it takes to complete the task when
instructions are delivered using AR as opposed to the other 3 delivery methods,
and use one-way ANOVA to test the hypothesis.
The
experiment represents a classic use of one-way ANOVA. The four different
instruction media serve as 4 categorical independent variables, and the amount
of time to complete the task serves as the dependent variable. In the Tang
experiment, an initial one-way ANOVA test shows a significant difference in
task completion time based on instruction media. The significant one-way ANOVA
result by itself, however, was not enough to prove the experimental hypothesis,
and the research team performed additional statistical testing to understand
the results. The reason one-way ANOVA was not sufficient is that it returns a
significant result when it finds a difference between any of the
independent variable group means and the aggregate mean. So, for example,
one-way ANOVA could return a statistically significant result if the mean
amount of time to perform the task using a printed manual was different than
the aggregate mean performance time, and this result would not have proven the
experimental hypothesis. Most statistical software packages do not report which
independent variable’s mean differs from the aggregate mean, so the researcher
must follow up either by plotting the results and/or performing additional
statistical tests to discover which independent variable(s) were responsible
for the statistically significant ANOVA result.
In the Tang
study, the research team conducted several post-tests, comparing task
completion time by pairs of instruction media. For example, the team tested if
there was a significant difference between task completion time in conditions 1
and 4 (printed manual vs. AR). The team conducted similar pair-wise analysis
for conditions 2 and 3 verses condition 4, and found the only significant
difference was between task completion time in conditions 1 and 4. Therefore,
the team concluded the results did not support the hypothesis; the amount of
time to complete the task using AR instruction was not faster than the other 3
instruction methods. The researchers did not mention which statistical test
they used to perform the post analysis. The recommended post-test for pairwise
comparisons after one-way ANOVA is Tukey’s HSD. A common mistake is to use a 2
sample independent t-test as a post-test for one-way ANOVA. Tukey’s HSD is
preferred because it is more robust than the t-test. With a t-test, the more
tests you run, the more likely you are to get a significant result just by
chance. Because Tukey’s HSD is more stringent, it decreases the probability of
getting a significant result by chance.
Could the
team have skipped using one-way ANOVA and just completed pair-wise analysis?
No, because some of the inputs for Tukey’s HSD are outputs from a one-way ANOVA
test, so running the initial ANOVA test is essential.
Another
caveat to note about one-way ANOVA is that it will not tell whether there is a
linear trend across groups. For example, the researchers might have
hypothesized the amount of time to perform the Duplo assembly task decreased
across conditions (e.g. performance time in condition 1 > condition 2 > condition 3 > condition 4). One-way
ANOVA would have been a proper starting point for the trend analysis. If one-way ANOVA delivers significant
results, the research team can follow-up by plotting the one-way ANOVA results
and examining them for an upward or downward trend. For a more robust
post-test, simple regression can be run to determine whether a linear trend is
statistically significant.
Values to report:
·
degrees of
freedom
·
F value
·
p value