One-way ANOVA

1 Continuous Dependent Variable with normal distribution

1 (Multi-level) Categorical Independent Variable

A one-way analysis of variance (ANOVA) measures whether one or more components of a multiple level independent variable predict the value of a dependent variable.

In ‘Comparative Effectiveness of Augmented Reality in Object Assembly’[12], Tang et al. test how long it takes subjects to complete a 56 step assembly task using Duplo building blocks. Subjects receive instructions for performing the task either via (1) a printed manual, (2) computer assisted instruction (CAI) delivered on a traditional computer monitor display, (3) CAI delivered on a head-mounted display, or (4) via Augmented Reality (AR). In the AR condition, subjects use a head-mounted display that overlays 3 dimensional instructions on the actual work space, including on the Duplo blocks. The research team hypothesizes there will be a significant difference between the amount of time it takes to complete the task when instructions are delivered using AR as opposed to the other 3 delivery methods, and use one-way ANOVA to test the hypothesis.

The experiment represents a classic use of one-way ANOVA. The four different instruction media serve as 4 categorical independent variables, and the amount of time to complete the task serves as the dependent variable. In the Tang experiment, an initial one-way ANOVA test shows a significant difference in task completion time based on instruction media. The significant one-way ANOVA result by itself, however, was not enough to prove the experimental hypothesis, and the research team performed additional statistical testing to understand the results. The reason one-way ANOVA was not sufficient is that it returns a significant result when it finds a difference between any of the independent variable group means and the aggregate mean. So, for example, one-way ANOVA could return a statistically significant result if the mean amount of time to perform the task using a printed manual was different than the aggregate mean performance time, and this result would not have proven the experimental hypothesis. Most statistical software packages do not report which independent variable’s mean differs from the aggregate mean, so the researcher must follow up either by plotting the results and/or performing additional statistical tests to discover which independent variable(s) were responsible for the statistically significant ANOVA result.

In the Tang study, the research team conducted several post-tests, comparing task completion time by pairs of instruction media. For example, the team tested if there was a significant difference between task completion time in conditions 1 and 4 (printed manual vs. AR). The team conducted similar pair-wise analysis for conditions 2 and 3 verses condition 4, and found the only significant difference was between task completion time in conditions 1 and 4. Therefore, the team concluded the results did not support the hypothesis; the amount of time to complete the task using AR instruction was not faster than the other 3 instruction methods. The researchers did not mention which statistical test they used to perform the post analysis. The recommended post-test for pairwise comparisons after one-way ANOVA is Tukey’s HSD. A common mistake is to use a 2 sample independent t-test as a post-test for one-way ANOVA. Tukey’s HSD is preferred because it is more robust than the t-test. With a t-test, the more tests you run, the more likely you are to get a significant result just by chance. Because Tukey’s HSD is more stringent, it decreases the probability of getting a significant result by chance.

Could the team have skipped using one-way ANOVA and just completed pair-wise analysis? No, because some of the inputs for Tukey’s HSD are outputs from a one-way ANOVA test, so running the initial ANOVA test is essential.

Another caveat to note about one-way ANOVA is that it will not tell whether there is a linear trend across groups. For example, the researchers might have hypothesized the amount of time to perform the Duplo assembly task decreased across conditions (e.g. performance time in condition 1 > condition 2 > condition 3 > condition 4). One-way ANOVA would have been a proper starting point for the trend analysis. If one-way ANOVA delivers significant results, the research team can follow-up by plotting the one-way ANOVA results and examining them for an upward or downward trend. For a more robust post-test, simple regression can be run to determine whether a linear trend is statistically significant.

Values to report:

· degrees of freedom

· F value

· p value