Broken windows, mediocre methods, and substandard statistics

Broken windows theory states that cues of inappropriate behavior like litter or graffiti amplify norm-violating behavior. In a series of quasi-experiments, Keizer, Lindenberg, and Steg altered cues of inappropriate behavior in public places and observed how many passersby subsequently violated norms. They concluded that particular norm violations transgress to other misdemeanors (e.g., graffiti leads to littering or even theft) and that the presence of prohibition signs heightens the saliency of norm violations, thereby aggravating the negative effects of cues such as litter and graffiti. We raise several methodological and statistical issues that cast doubt on Keizer et al.’s results. Problems include confounding factors, observer bias, lacking scoring protocols, a failure to establish interobserver reliabilities, inflated Type I error rates due to dependencies, sequential testing, and multiple testing. We highlight results of a highly similar study that does not support the notion that prohibition signs aggravate the effects of observed norm violations. We discuss potential improvements of the paradigm.
Keywords: descriptive norms, littering behavior, methodological critique, statistics, theft