Binomial test

1 Bi-level Categorical Dependent Variable

0 Independent Variables

A one sample binomial test measures whether the proportion of two categorical dependent variables significantly differs from a hypothesized proportion. [13]

A 1989 study performed by William Cole uses the binomial test to measure the results of an information visualization experiment [1]. Cole began by demonstrating humans have difficulty performing Bayesian reasoning. To illustrate the problem, Cole cites a classic case requiring Bayesian reasoning. The case is as follows: (1) physicians correctly diagnose a given disease 95% of the time, (2) 90% of the time, physicians correctly rule out the given disease when it is not present, and (3) the disease is present in one person out of a thousand. Given this example, what is the probability a person, chosen at random, who tests positive for the disease, actually has the disease? Most people, including Harvard physicians, answer incorrectly, estimating the probability at 90% or higher, when the correct probability is actually closer to 1%. Cole developed visual displays he believed would help humans to better comprehend Bayesian reasoning problems such as this one.

In the experiment, test subjects, all of who demonstrate difficulty performing a Bayesian reasoning task in a pre-test, learn to use Cole’s Bayesian reasoning visual display. After using the visual display, subjects take a second Bayesian reasoning test. Cole uses the binomial test to measure whether the subject’s score on the post-test improves after using the visual display, and finds it does.

The post-test had just one question, and there were only 2 possible answers to the question, one right and one wrong. Cole tested whether subjects who learned to use his graphical visualization display chose the correct answer more than 33% of the time, where 33% was the binomial test hypothesized value.

Cole’s study demonstrates a valid use of the binomial test. First, the dependent variable, the post-test score, was expressed in categorical terms; either the subject got the answer right or wrong. Further, the dependent variable was recorded as a proportion (percent right versus wrong) and compared to a hypothetical proportion. One piece of information Cole does not report, but should have, was the reason he chose 33% as his hypothesized proportion. The lack of explanation does not invalidate the test, but without it, the reader is unsure what claim can be made with respect to the test results. Cole’s experiment is a good example of why the binomial test is not used very often in HCI research; there aren’t very many situations where researchers need to compare a dichotomous dependent variable against a hypothesized value.

Values to report:

· the hypothesized value, and why it was chosen

· percent of values satisfying the condition (e.g. actual percentage of correct answers)

· p value