The COMPARABLE Checklist (suitable for printing!).
This checklist can be applied to any numerical comparison, although not all items apply to every comparison. This is an abbreviated version of the checklist; the chapters of the text describe these elements in more detail.
C: Were appropriate comparison groups picked? Was like compared to like?
O: Were the terms of comparison operationalized/defined fairly, and were accepted and appropriate measures chosen?
M: Run a mental experiment: Do the conclusions make “back of the envelope” sense?
P: Are the pictorial/graphical representations unbiased and appropriate to the comparison?
A: Which factors were accounted for/controlled for (inflation, population, income) and which were not?
R: Could the difference be nothing but randomness at play?
A: Did you consider alternative measures of change/gain/difference (flows vs. stocks, percentage vs. percentile)?
B: Were base rates taken into account? Did we see both absolute and relative increases?
L: Would a longitudinal/cross-sectional analysis tell a different story?
E: What is the story of the edges? What is the story of the center? How are they different and what does that mean?