Using data from Wolfram Alpha or some other source, replot the NASA graph here http://ripetungi.com/the-rise-and-fall-of-nasas-budget/ either:

1. Controlling for inflation,

2. Tracking expenditure as a percentage of GDP, or

3. Tracking expenditure as a percentage of government spending.

Defend your choice, and note any differences in the shape of the graph that result.

Questions:

• What is wrong with this graph? In what ways is it deceptive?
• How would you fix it? What might be a better format?

So is it something based on what they sell? Nope. It largely comes down to confounding factors.

Do a small amount of research on the clientele and locations of these stores, and identify some lurking variables that may account for these differences.

Second, consider this poll from The Washington Post’s website. Walk through what would happen if you set up the poll in various different ways. Be ridiculous where it helps. For the moment, ignore issues of non-response and response bias.

Name at least one strength and one weakness to setting up the options in the way they have.

Norfolk (VA) and San Francisco (CA) both have approximately the same average temperature (57 degrees Fahrenheit). But that temperature differs month by month.

Questions:

1. Looking at the above graphs, which of the two cities has the highest standard deviation in monthly temperature?

2. People generally use heat and cooling technology to keep the temperature of houses above 65 degrees, and below 70 degrees. In which city are you likely to spend the most money on electricity and fuel for heating and cooling? How does this relate to standard deviation?

Ninety-four percent of students who earn a bachelor’s degree borrow to pay for higher education — up from 45 percent in 1993, according to an analysis by The New York Times of the latest data from the Department of Education. This includes loans from the federal government, private lenders and relatives.

From The Chronicle:

Nearly 53 percent of full-time undergraduate students borrowed money to attend college in 2007-8, compared with 49.5 percent in 2003-4, according to the Education Sector’s analysis of data from the National Postsecondary Student Aid Study.

From FinAid.org:

Few students can afford to pay for college without some form of education financing. Two-thirds (65.6%) of 4-year undergraduate students graduated with a Bachelor’s degree and some debt in 2007-08, and the average student loan debt among graduating seniors was $23,186 (excluding PLUS Loans but including Stafford, Perkins, state, college and private loans).

Assume all of these depictions of the current state of student loan debt are true. Why is the NY Times estimate so high? Why is the Chronicle estimate so low? Look at issues associated with “O” (How was the variable Operationalized/Defined?) to provide some answers.

What information would you like from the NY Times to evaluate its high figure? Consider issues of “E” (What is the story of Edges & Subpopulations?)  in thinking about the meaning of the figure.

This chart could mean that the more education you get, the more you drink. What is another explanation for the increase in average annual expenditures on alcohol?

A Note to Instructors

We’ve broken this text up into chapters. In our estimation, each chapter could take one to two weeks depending on how much time in your class is dedicated to other activities and needs. A four-hour, twice-a-week-class with no other content or class-time obligations should be able to do a chapter a week.

There are six chapters of bootstrapping into basic skills, followed by an appendix which deals with various more specific issues. The idea is you would teach the first 6-10 weeks of a class with the main sequence, then build your own second half of the class out of either the materials in the appendix, a class project, or an additional topical supplementation.

We feel, however, that it is important to go through these initial six chapters in sequence.

• Which intersection in town is the most dangerous?
• How much more expensive will college be if I graduate a year late? 
• Which product line has given our business the best overall return in the past two years? 
• At what rate is global temperature rising?
• How expensive was the most recent election compared to previous elections?

Comparisons don’t happen in a vacuum. Usually when someone is comparing things, they are comparing them for a reason. In the case of the intersection question above, maybe there is an action pending – if we are going to upgrade one intersection, which one should it be? Businesses may want to know what products have been the most profitable so they can pursue profitable avenues at the expense of the less profitable ones. A political scientist may be investigating the influence of money on elections, and trying to determine if that influence has increased over time.

Ultimately, comparisons have real world consequences. If you rightly determine which intersection is the most dangerous as an urban planner, perhaps you can save a life. Knowing which product lines have given a company a good return could be the key to keeping a business afloat, saving your job and the jobs of others.  Determining whether money in elections is out of control or in line with historical trends can help us plot a course of action for our country that fixes what is wrong with our system while preserving what is right.

Depending on what profession you go into, you may use algebra or you may not. Some of you may use calculus or trigonometry. Some of you will be asked to use advanced statistical methods. Most won’t.

But every single one of you will be asked to compare things as an employee, consumer, and citizen. And whether you are able to compare things adequately will have dramatic effects on the success of your business, your family, and your community.

This is a book about how to use very simple statistical techniques to compare things. It is not so much about formulas as it is about critically thinking about numbers. We honestly believe this skill will be one of the most important skills you acquire in your college career.  Mastering it will change your life for the better, and get you closer to being the sort of person you want to be.

What we hope you will develop by the end of this course is the ability to critique a comparison (and perhaps suggest alternate comparisons) in a ten minute time-frame.

Initially, your projects will take longer than ten minutes. At first they may take much, much longer. But by the end of this course you will be able to do in ten minutes what it would take people hours to do. At the end of this course, presented with any comparison, you’ll be able to deconstruct it methodically, evaluate it, and put it back together again in the time it takes to brew a pot of coffee.

Keep in mind as you work through the activities that this is the goal. It won’t be a ten-minute endeavor right off the bat — we do things that are long and difficult so that they become short and easy. But the goal is ultimately the sprint.

The Fair Comparison Checklist (COMPARABLE) is meant to provide you a quick way to break down a comparison and critique it.

As we go through the course, the categories presented here will have more meaning. For now, we want to introduce the checklist, and, if we can beg your indulgence, ask you to memorize it.  Memorization has fallen out of favor in some educational circles, but recent evidence indicates that, even in a world where information is a smartphone away, people who remember key facts, checklists, and processes perform substantially better than those who don’t.

When information is stored only externally, when you have to retrieve it from the web, learning is likely to only happen when you consciously initiate it. If something rises to the level of pulling out your smartphone and firing up Google, you will learn, but otherwise not much gets connected.

This is a shame, because when we can rely on memory, our minds are learning machines. Facts and concepts stored in memory can be triggered contextually, without effort, as we go about our day.  And every time that fact, concept, or process is triggered or recalled, more connections are made, and learning deepens.

Remembering is a fundamental learning process. If we do not exercise our memory and try to make new mental connections, we cannot expect to grow as learners.

It’s our hope that by taking the twenty minutes to memorize this list now, you will save yourself dozens of hours over the course of the semester in studying. By memorizing the list you’ll be able to reinforce this learning throughout your day, without pain, instead of only at your desk.

So, without further ado, here’s the list. You will not understand every term right now. For now, just memorize the nouns highlighted by the mnemonic, e.g. C=”comparison groups”, O=”operationalized/defined”, M=”mental experiment”, etc.

C:  Were appropriate comparison groups picked? Was like compared to like?

O:  Were the terms of comparison operationalized/defined fairly, and were accepted and appropriate measures chosen?

M:  Run a mental experiment: Do the conclusions make “back of the envelope” sense?

P:  Are the pictorial/graphical representations unbiased and appropriate to the comparison?

A:  Which factors were accounted for/controlled for (inflation, population, income) and which were not?

R:  Could the difference be nothing but randomness at play?

A: Did you consider alternative measures of change/gain/difference (flows vs. stocks, percentage vs. percentile)

B: Were base rates taken into account? Did we see both absolute and relative increases?

L:  Would a longitudinal/cross-sectional analysis tell a different story?

E:  What is the story of the edges? What is the story of the center? How are they different and what does that mean?

And that’s it. Ten questions that will eventually allow you to critique any comparison. The rest of this course is about understanding what these questions mean and how to apply them in practice.

Memorization is a lost art. Most students memorize by “cramming”, and believe that memorization does not work long-term. But memorization can work long-term – it is only cramming that does not work.

So how do you memorize something like the list above?

The method below takes what we know about how the brain learns and applies it to this specific task. To memorize this whole list, complete with the full sentences, is going to take you about an hour and 15 minutes, but it is going to be spread out over the space of a week.

Here’s what you do:

DAY 1:

1. Read the list through and try to understand as much of it as you can. Look up words you don’t understand. This is not memorizing yet – this is just deep reading. Be methodical, but don’t waste time now trying to commit it to memory. If you have read through the list two or three times and thought about it, it’s time to move on.
2. This is where the memorization begins. Take a chunk of it, say the first four letters: COMP (Comparison Groups, Operationalize/Define, Mental Experiment, Pictorial/Graphical Elements).
3. Draw it out on a piece of paper or a whiteboard. Start from the center: write COMP, and circle it. Draw separate arrows out to the other words (Comparison Groups, Operationalize/Define, Mental Experiment, Pictorial/Graphical Elements).
4. Make mnemonics – for “Operationalize” draw a picture of someone having an operation, or of the game Operation. Put it right next to your word operationalize/define. Next to “Pictoral,” draw a graph, or even better, a picture of the Mona Lisa. Put a “p” on it. Next to “Comparsion Groups,” draw a group of short stick figures standing next to tall stick figures.
5. Now with the book open, walk through the phrases using the sheet. Look at the picture you drew for “comparison groups”, and read the text associated with it “Were appropriate comparison groups picked? Was like compared to like?”. Write it down next to those stick figures you drew. Do the same with the other words.

Now here’s the important part: Throw this drawing away. Take one look at it, and throw it away. Close your book, and try to reconstruct it from memory.

You’ve just discovered the first important lesson about memory – assuming you’ve done the reading to begin with and have some understanding of it, you learn best with the book closed. You’ll learn more by trying to remember what you read than by reading it a dozen times.

You should be able to do everything above in about 10 minutes. Check your answers. If you’ve got more than 10% of it wrong, close the book, throw out your drawing, and try again. Remember to talk while you draw, and explain what you are doing.

This part will take an additional 5-10 minutes.

Now you should be able to reproduce this diagram that you’ve made. Bring a piece of paper and a pen around with you, and over the next 24 hours, check a couple of times that you can reproduce the full diagram quickly from memory. Sketch it out in 60 seconds before dinner. Rehearse the associated phrases. Again, if you can get 90% of the wording right, move on.

Save the last drawing you make for later reference. Make sure it is all filled in.

DAY 2:

We have an odd goal today. We want to make you forget what you learned yesterday on DAY ONE, or nearly forget it. Why? Because learning deepens when remembering is hard, and remembering your COMP drawing has gotten too easy.

This is the second lesson about memory. We learn by almost (but not quite) forgetting. So do not think about the COMP part all day.

Today you are going to memorize the ARA part of the list. And our hope is that learning that will interfere a little with your COMP memory.

So take the ARA part of the list, and do the same thing we did above. Draw ARA at the center. Maybe “Accounted for” has accounting books next to it, or it has Harold Ramis, the accountant in Ghostbusters. Or the Count from Sesame Street (“A Count”, right?). Follow all the steps until you can walk through the diagram and say the associated sentences at about 90% accuracy. Remember to study with the book closed.

Repeat the steps above – try to draw the diagram and talk-through or write the associated sentences a couple of times in the day. Remember to save your final drawing for later reference.

DAY 3:

On Day 3, you are going to try not to recall either COMP or ARA from the COMPARABLE mnemonic.  We’re just going to work on the BLE segment, the same as above. Come up with something for Base rate, Longitudinal/cross-sectional, and Edges/Center. Repeat the same steps as above.

DAY 4:

Take Day 4 off. Honestly, don’t think about any of this. (Unless this happens to be a class day, in which case you have no choice!)

DAY 5:

Now try to put it all together. Hopefully you’ll struggle a bit with remembering pieces – if you struggle, that means your mind is coming up with new connections which will make this easy in the future. Struggling to remember something is like feeling muscle fatigue at the gym – as long as you can eventually pull off the memory task, the fact it was difficult is good – it means you were pushing your mind to its limits, which is how you are going to get that memory better indexed.

So do the whole COMPARABLE word, complete with writing out the full sentences, drawing the pictures, arrows, etc.  Talk as you do it. If you get stuck, do not immediately consult the sheet to see what you forgot – work through the difficulty of remembering. Can you remember what the picture was? Can you visualize the rest of the sheet in your mind?

Moments of forgetting are gifts to those trying to memorize things. DO NOT WASTE THEM by immediately consulting this book or your cheat sheet. Work through it.

DAYS 6-INFINITY

Once you can reproduce the drawing with 90% accuracy, you only need to draw it and talk it through every two or three days for about one to two minutes. Just do it when you have a down minute or two. Pick a couple random letters from the acronym, shuffle the order, and do just those letters, say, A1, R, E. Do it on paper, or in your head while waiting in the Dining Commons line.  As long as you can do segments like this in under a minute or two, you don’t need to study this acronym further.

And that’s it. That’s the memory technique which will make your college life easier.

There are some general principles you can take away from here that apply to all studying:

• You do your best studying with the book closed.
• Working hard at studying feels like struggling to remember.
• Use pictures to remember, even if you can’t draw.
• Study in small sprints over time, with intermittent reinforcement – marathons will get you nowhere.
• Focused, solo effort will get you further than study groups.
• Study smart to spend less time studying!

Final Note

Just a note on that last bit about studying – you may think I don’t know what I’m talking about. But ages ago, as an undergraduate, I was on academic probation. I felt like I was working, at least sorta-kinda working. Working enough, at least. I had numerous things always hanging over my head. But after my GPA fell to well under 2.0, I left school.

When I came back, there were some stipulations. My parents had me take a 21 credit course load. I had to maintain a semester average GPA of 3.0. I had to graduate in three semesters.

For the first time in my life, rather than just study, I decided to think about what the most efficient way to study would be. When I got a textbook at the beginning of the semester, I’d quickly flip through it and look for any hooks I could start memorizing early. I’d read chapters a couple of weeks before they were due, and then quickly review them before class a couple weeks later.

It wasn’t all roses – I still procrastinated on papers and probably made too many last minute changes to projects I worked on. That’s my personality, and I’m stuck with that to some extent. But to my surprise, when approached methodically, the time I spent on classwork went *down*. And my GPA? It skyrocketed to a 4.0 – in a 21 credit semester full of upper-level courses. And it stayed a 4.0 every semester since then, with the exception of one semester in graduate school where it ended up a 3.8.

Studying smart doesn’t mean studying longer – it means studying harder and better so that you can do well and still have time for other things in your life. It’s the difference between going to the gym and lollygagging on an orbital machine for an hour versus doing interval training and getting out in 20 minutes.

Hopefully you have tried to use the method to memorize COMPARABLE that we describe above. You really should – when you see this work, it’s like a magic trick. But even if you don’t do that, please think about the way you use your study time, and work to use it more effectively. Take the time to look up note-taking tips on the web, or memorization techniques. Look at some of the research on how to do deep critical reading of texts that gets beyond surface understanding quickly and efficiently. Work smart so you can work less.

• C: Were appropriate comparison groups chosen? Was like compared to like?
• A1: Which factors were accounted for/controlled for (population, inflation, income) and which were not? 

There are many rules that apply to making fair comparisons, but there is one rule that leads, in one way or another, to most of the other rules:

Compare Like to Like When Possible, and Account for Things When Not.

Why? Because usually when we compare things, we are interested in the difference that one thing or factor makes.

For instance, if we are comparing laptops that we might buy, we’d probably be comparing relatively similar laptops, and we might start with price. Laptop 1 is $500 but Laptop 2 is $600. Price is the measure we will use to do this comparison.

So Laptop 2 costs more than Laptop 1, but is that the end of the story? If they are exactly the same laptop, then likely it is. But they are probably not the same. Looking closely, you might find that the more expensive laptop has 2GB more memory, and 300GB more hard drive space.

To make the decision of which one to buy, you need to be able to compare like to like, or apples to apples. Given that the differences between the two laptops are the amount of memory and hard drive space, you might ask yourself how much you are willing to pay for those additions. Would you pay $90 for the extra hard drive space? And for the extra memory, would you pay $30?

Laptop 2 includes the extra memory and space and costs $600. Laptop 1 does not, and costs $500. For Laptop 1 to be comparable to Laptop 2, you’d need to add in the extra memory and space.

You’ve already decided the value of those additions, so you can add that amount to the cost of Laptop 1: taking into account the features I want, the real price of Laptop 1 is $500 + $90 + $30, or $620.

Now you can compare like-to-like: Laptop 1 (when the features are taken into account) is $620 and Laptop 2 is $600. In this case, assuming you have accurately judged what the features are worth to you, Laptop 2 is the better deal, even though it initially seemed more expensive. And the way we were able to find that out was by using a mathematical procedure to account for the differences.

What is true of shopping for laptops is true for most comparisons. We try to pick similar things to compare, but when that is not possible, we use math to account for the difference.

HEALTH CARE

Here’s another simple example – say we are interested in how effective the U.S. health care system is, and someone gives us the following opinion supported by a comparison:

Of course we have good health care here. The average life expectancy in Africa at birth is 54 years. In America it is 78 years. I don’t understand what all this fuss is about. Seventy-eight is much bigger than fifty-four!

What’s the problem with that?

Well, first of all, Africa and the U.S. are not the same, so we are not comparing like to like. Africa is much poorer than the U.S., and national wealth generally predicts life expectancy. Africa also deals with a variety of infectious diseases like malaria and AIDS that are far less of a problem in the U.S.  Many nations in Africa are in a state of war or engaged in other military hostilities, and many deaths result not from disease, but through dying in battle or, as in Rwanda, dying through genocide or purges.

Given all of this, we would expect Africa’s life expectancy to be very low– and the reasons why it is low have very little to do with the effectiveness of health care, as your friend had argued. Instead, the shorter life expectancy in African has to more with war, limited resources, and widespread infectious disease.

Frankly, this isn’t a “laptop-to-laptop” comparison – there are far too many variables here with such a great impact that a different comparison should probably be chosen. We can’t possibly account for all these factors in a meaningful way.

So what we would want to do is chose a country very similar to the U.S., or better yet, choose a bunch of countries similar to the U.S.  One very simple way to do international comparisons of this sort with America is to use the so-called G8 economies – a set of 8 countries that have some of the largest, wealthiest economies in the world. Figure 1 shows what life expectancy looks like for those countries when we search Wolfram Alpha for “Life Expectancy G8”.

When we do that we find that, with the exception of outlier Russia, the U.S. ranks lowest in life expectancy among the G8 countries.

Of course, what this shows us about the health care system is debatable. If we are concerned our system is not as efficient as other health care systems, we might check to see whether these countries are spending the same amounts on health care. This is called “controlling for spending.” If we think that America is different due to personal behavior or genetics, we might control for obesity, or compare subpopulations according to ethnic background.  But by starting with something that was at least broadly comparable, we are on the right track.

COLLEGE COMPLETION RATE

Politicians, parents, and students are often concerned with the completion rate of a college.  How many students that start at a given school complete in four years? How many in six? And so we come up with comparisons: College A has a four-year completion rate of 71% whereas College B has a four-year completion rate of 53%.

Obviously College A is your better bet, right? Go there and you’re more likely to graduate in four years!

Well, no.  Think for a minute and you’ll see why. What are we trying to figure out?

In the case of a parent or student, you are probably trying to figure out how easy the college makes it to graduate in four years. Do they offer enough seats in classes? Are the required courses reasonable, and offered on a predictable schedule? Does advising assist students in making the right choices?

In other words, you want to know the difference the college makes in terms of helping you graduate.

But college is not the only factor at work here. A college that accepts primarily wealthy students may have a higher completion rate – but that could be because none of its students had to drop out due to lack of money, or because most of the students that go there are rich enough that they don’t have to work part-time jobs while attending college.

If that’s the reason why the difference exists, it’s not going to do you any good. Changing your college will not change whether you have to work, or make your parents richer. So what you want to do is to do the same thing we did with the laptop and the health outcomes. You want to take the impact of the student’s wealth out of the equation so you can compare like-to-like.

In this case, we can use what we call subpopulations to help us out. Let’s assume you are a Pell eligible student (which is generally a marker indicating moderate to low family income).

Instead of comparing the entire population of College A to the entire population of College B, we can compare the Pell-eligible subpopulations by asking a slightly different question. Instead of asking what the completion rate at both colleges is, we ask what the completion rate for Pell-eligible students is at each college – in other words, we find comparable subpopulations so that we can compare like-to-like.

When we look at that, we may find that College A is worse. How is that possible? Consider the following scenario:

Read through the different parts of this chart until you understand what’s going on. Notice that College A does worse than College B at both educating Pell-eligible students and at educating non-eligible students. Yet College A does better, and significantly better, than College B in four-year graduation rates. Why?

Simply put, it’s because College A accepts fewer Pell-eligible students. In other words, the main thing that is influencing four-year completion rates is not the school itself; it is the proportion of wealthier students at the school.

College A has a dismal record with graduating Pell-eligible students on time. But these students only comprise a half of one percent of their population. The majority of their students are faster-graduating non-Pell students, so their rates look good.

By breaking it apart, and making a like-to-like comparison, we can see that College B does at least as well as College A with the wealthier subpopulation, and does significantly better with the Pell population. Once we compare like-to-like, the real story emerges.

If you don’t understand what is going on in the chart above, work on it until you do. This is one of those big bang-for-the-buck insights worth spending some time on.

We will talk about this a bit later in the book when we discuss a thing called Simpson’s Paradox. For the moment, though, just remember: Compare like-to-like when possible, and account for things when not.

This section deals primarily with the following elements of the COMPARABLE framework:

• E:  What is the story of the edges? What is the story of the center? How are they different and what does that mean?

We’ve been looking at various comparisons, in many cases using what statisticians call measures of central tendency.

There are three common measures of central tendency:

The mean is the total of all items in a set, divided by the number of items in the set. It is what most people are referring to when they talk about the average.

The median is the “middle number” in a set that is ordered from smallest to largest.  For instance, in the set {2,2,2,3,3,4,5}, three is the median. In the case that there is an even number of numbers in the set, the median is the average of the the two middle numbers. In the set {1,2,3,4}, 2.5 is the median.

The mode is the number that appears most in the set. In the set {1,3,3,5,5,5}, five is the mode. A set like {1,2,2,5,5,6} is bimodal – it has two modes. A set like {1,2,3,4,5} has no mode.

All of these measures are ways to quickly grasp the “tendency” of a set of numbers, and ultimately the reality of the world they represent. That sounds abstract, but it’s actually quite simple.

Imagine, for example, you are trying to figure out how much you are likely to spend in college on textbooks. That’s going to vary widely – and the best way to figure it out might be to ask a bunch of students you know how much they spend on textbooks. Doing that, you might get a range of answers like this:

{133,273,253,352,440,229,64,219,94,130,360,132,132,178,340,314,55}

The graph and summary statistics in the figure show what this set of data looks like.

In this case, the mean and median are pretty similar. If you are budgeting for your own books, you might expect to pay about $217.50 (the standard deviation indicates that might be risky, but we will deal with that at a later point). The median tells you that you might budget $219. These values are pretty close, and in this case the mean might be the better representation – it’s slightly lower price takes into account the couple of sub-$100 classes a bit better.

But let’s make a change. Let’s imagine a world where most of your friends pay over $200 for their textbooks, but a couple of them pay nothing.

Why? We don’t know. Maybe they are lucky, or in a major with no texts. Maybe they use Open Educational Resources, like Khan Academy and MIT OpenCourseWare. In any case, here’s the distribution:

{0,0, 219, 229, 233 ,232,232, 240,253,252, 260, 273, 278, 314,330 }

This is where things get interesting. Look at the mean: $223.  Now ask yourself: out of 15 students I asked, how many paid $223 or less?

The answer? Three students.

Now how many of the 15 paid $223 or more?

The answer? Twelve students.

So which of these is more representative of the experience of students? Can we really say a figure like $223 is a good representation if 80% of students pay more than that?

No, of course not.

So let’s look at the median. By definition, the median has as many items above it as below it. So in that way, it’s pretty accurate. In this case, it’s also close to a lot of the values. It’s relatively unaffected by the $0 students, but in this case that is probably a good thing, as we saw in the mean example. So if you were going to prep for a price, $240 would make a lot more sense than $223.

Again, because we are working with a very small data set, this is an exaggeration to some extent. There’s a lot of randomness and variation here, and you are probably going to want a bit more buffer in your budget. We’ll come back to this example later. But the point is in this particular case you probably would start with the median as an assumption and not the mean.

When we return to this issue in later chapters, we’ll talk about this in a bit more detail. But when looking at something like this, the question you should be asking yourself is which measure of central tendency best represents “normal” experience, given the range of numbers in the data set.

THE STORY OF THE EDGES

Right now, let’s look at another question our textbook analysis brings up – what’s up with those zeroes?

The answer to this could matter quite a lot to your budget. If the people who paid nothing for their textbooks are more similar to you in some way, our Golden Rule (Compare like-to-like!) kicks in, and we are going to have to weight those zeros more prominently in our analysis of how much we will pay. If the people at the edges differ from us in ways that impact textbook price, we may want to discount those items in our set completely.

In this case, let’s say we dig down and find out from the two people that paid zero that they are both students taking one class online, and the lack of textbooks is due to the online resources built into the class.  When you look at the other students you asked, the ones that paid more than zero, you find that they are all taking face-to-face courses.

So do the two “zeroes” in the data help you predict your textbook cost? Will their book prices help you guess your own? In this case, if you are a student taking a small number of online classes, the answer is yes – their cost may be more pertinent to you than the other 12 students, especially if online resources are a norm in the online classes at your college. On the other hand, if you are in a face-to-face program, those classes may not be relevant to your situation at all. In fact, you might want to remove them from your sample entirely.

Dropping the zeros, in this case, and only looking at textbook costs for face-to-face instruction, we find a different story than before. The median jumps up from 240 to 252, and, more notably, the mean is now *higher* than the median, being pulled up by the couple of textbook costs over $300.

Here, again, we are likely to choose the median value, given the difference between the median and mean.  But the important point is by investigating the story at the “edges”–those values at the extreme ends of the data set–and reformulating our question from “How much are students paying for books?” to “How much are student in face-to-face classes paying for books?” we probably significantly increased the accuracy of our prediction, and found that books were much more expensive than our initial data indicated.

LIFE EXPECTANCY IN THE U.S. AND FRANCE

Of course, you are unlikely to put together a table of data to figure out the estimated price of your school textbooks. Normally you are working from summary statistics you are given that were compiled by other people.  And you don’t have access to the data, so computing the mean and median may not be an option.

Let’s take a really simple but important question. How long can you expect to live?

Life expectancy, like a number of other common statistics, is not distributed equally around a center. Instead it demonstrates what we would call a strong left skew. Here is a chart from Wolfram Alpha showing when people in the U.S. die:

The mean age of death in the U.S. (at birth) is 78 years. The median age of death, on the other hand, is 81.9 years. And it would make sense to use that here, because we can see that the situation is similar to the initial “textbook price” situation.  The spike of deaths before age one (look carefully at the left edge of the chart) act like the zeroes in our textbook example and pull our mean down. If we are looking to understand a normal age of death in the U.S., putting much emphasis on these  numbers at the far edges of the data set doesn’t make sense.

What’s more fascinating than the median, though, is the range around it. If we take the middle 50% of people (people who lived longer than the bottom quarter of the population, but shorter than the top quarter) you’d find a wide range of age of death. We call this “middle 50” the interquartile range, and here it starts at 71, but runs almost up to 90.

Where this becomes really interesting is when we make comparisons. Consider this comparison of life expectancy in the U.S. with life expectancy in France.  We can start by comparing means, as imperfect as they may be – the U.S. has a mean age of death of 78, whereas France has a mean age of death of 81.

What about the median? We can see immediately that France’s median age is three years higher than the U.S.’s median age. But perhaps more interestingly, we see that the interquartile range is smaller – the middle 50 in France die within 16.4 years of one another.  In America, that stretches to 18.4. In other words, at least by this measure, there is more variation in age of death in America, and we might be interested in why that is. Is it due to differences in health care? Genetics? Variation in lifestyle?

Other differences stand out as well. For instance, a quick comparison of the top 1% and bottom 1% in France and the U.S. shows the top 1% in the U.S. actually live longer than the top 1% in France. But on the bottom of the distribution it’s a different story. By the age of 25, 1 in 100 people in France have died. But in the U.S. 1 in 100 people have died by the age of sixteen. If you’re looking to explain the difference in mean life expectancy, you’ll find some of the answer in infant and child mortality. The U.S. has a lower mean life expectancy, at least in part, because a bigger percentage of our infants and children die.

We are interested in variations from averages partially because by comparing different subpopulations against the average, we can extract more meaning from our comparisons. The measures of central tendency provide starting points for conversation, but it is digging down into the specifics of the distribution that often provides the most useful insights.

The point is that measures of central tendency, like means and medians, raise questions which can only be answered by looking at distributions. When we ask for both the “story of the center” and the “story of the edges” this is part of what we mean.

1. Consider the following quote:

We often talk of social statistics, especially those that seem as straightforward as age, as if a bureaucrat were poised with a clipboard, peering through every window, counting; or, better still, had some machine to do it for them. The unsurprising truth is that, for many of the statistics we take for granted, there is no such bureaucrat, no machine, no easy count, we do not all clock in, or out, in order to be recorded, there is no roll call for each of our daily activities, no kindergarten 1, 2, 3.

What there is out there, more often than not, is thick strawberry jam, through which someone with a bad back on a tight schedule has to wade—and then try to tell us how many strawberries are in it.

–Blastland and Dilmot’s The Numbers Game.

What part of the COMPARABLE framework does this quote deal with?

Write down your answers and bring them to class. We will review your answers either in class or in a net-mediated peer instruction activity. You may also be asked to turn them in for grading.

1. Read this news report about a study that compares the cancer rates of women that eat large amounts of citrus and those that do not.  Why is this not a fair comparison? Name 4 to 5 things that you would want to account for in order to make the comparison fair.
2. In this editorial, David Brooks claims many people that compare American government spending to European government spending are not comparing like-to-like. What do you have to include to do an apples-to-apples comparison? Why is it important?
3. Look at this graph, comparing global air temperature from year-to-year. It appears that the eruption of Mount Pinatubo had a cooling effect on global air temperature, possibly due to the ash it kicked into the atmosphere. If we were to control for the eruption of Mount Pinatubo, how would this graph likely change?
4. Create a chart like the “Pell-eligible” chart from the college completion rate scenario. For your example, show how the overall patient mortality rate at Hospital B could be higher than Hospital A, even though Hospital B has lower mortality for both patients arriving critical and non-critical condition. Make the numbers up, but be sure they support the scenario.

Remember — questions will not only test the previous chapter. They will test all previous content (In this case everything from the golden rule, to subpopulations, to averages, to distributions). Figuring out which concept to apply to each problem is often the crucial skill, so we don’t indicate which concept comes into play.

In some cases you may have to look up a term or two. Look it up!

In most cases you may have to apply some creative thinking as well.

1. Sweden has a higher life expectancy than the U.S., but also a higher death rate: in America 8 people per 1,000 die per year, but in Sweden 9 people per 1,000 die per year. This is actually a pretty significant difference. Give a reason that can explain how a country with a high life expectancy can also have a high death rate. In other words, what can we guess about Sweden from knowing that they have both a higher life expectancy and a higher death rate than the U.S.?
2. The average shoe size (measured in inches) of a K-5 teacher is significantly smaller than that of a college professor. Why?
3. The average outstanding student loan balance per borrower in the U.S. is $23,300 (2010). The median balance is $12,800. Draw a toy density curve of what student debt might look like. Be sure to label x and y axis and demonstrate skewness.
4. Patients that get chemotherapy for cancer die at a much higher rate than patients that don’t. A naïve explanation might be that chemotherapy kills people, or, at the very least, doesn’t work. Give an alternative explanation.
5. In Africa the U.N. estimated AIDS prevalence by testing blood samples of pregnant women undergoing neonatal tests. Assuming nothing was accounted for, do you think this method would underestimate or over estimate AIDS prevalence of the general population? Of women? Why?
6. In New England, almost all college-bound students take the SAT. In the Midwest, students that are going to less selective schools take the ACT, and students going to more selective schools take the SAT. Which region will have the higher median score?
7. The average SAT score in NH in 2011 was 1559. In 1999 it was 935. Do some web research and explain the difference. If you were going to compare 2011 scores to 1999 scores, what would you do? What other factors would you take into account?

After completing chapter one, you have the basics of the COMPARABLE framework.

Week Two will introduce you to time-based comparisons, term definition, the art of performing mental experiments, and the many uses of percentages.

This section deals primarily with the following elements of the COMPARABLE framework:

• M: Do a mental experiment.

In the process of writing this book, I’ve come to the conclusion that one of the most important parts of any comparer’s toolkit is hypothetical thinking.

Hypothetical thinking comes in many forms, but the key skill is gaining a better understanding of a problem by asking “What if X were true?” and working through what the impact of that might be.

Using Toy Models on the “Mancovery”

A model is a simplified, scaled-down representation of a larger object or process.

Just as a model car does not represent every detail of a car, an intellectual model does not represent every detail of a situation. The idea is to make things comprehensible by removing the distracting complexity.

Still, mathematicians and statisticians create very complex models. The models that predict weather and global warming, for example, contain hundreds of variables in complex, interwoven interactions. So to stress the “sprint-like” nature of what we mean when we talk about models, we’ve chosen the phrase “toy models” to highlight how small these can be.

What does a toy model look like? Consider this article on whether the current economic recovery is a “mancovery”:

Men, who lost more than twice as many jobs as women during the worst economic slump since the Great Depression, have landed 88 percent of the non-farm jobs created since the recession ended in June 2009. The share of men saying the economy was improving jumped to 41 percent in March, compared with 26 percent of women, according to the Bloomberg Consumer Comfort Index’s monthly expectations gauge.

 

“The recovery is a mancovery,” said Heather Boushey, a senior economist at the Washington-based Center for American Progress. “I don’t see improvement for women in the past year, whereas for men this is the best year in years.”

 

So here’s the question — if men lost 100% more, or twice as many, of the jobs in the recession than women, what percentage of new job openings would we expect them to get back if we were looking for an equitable recovery? Is it above 88%? Below 88%? What is the exact percentage?

You might know the answer to this already, but, if you don’t, you can do a Mental Experiment. Plug in some fake numbers and find out!

Here’s how that would look:

Mental Experiment: “Mancovery” from Mike Caulfield on Vimeo.

So the 88% does, to some extent, represent a “mancovery”, though maybe not by the amount it initially seems (66% of jobs going to men would be equitable, so this is about 33% more jobs than we might expect).

Could you have figured this out without an experiment? Absolutely. An easy way to look at this is that if men lost double the jobs, they must have lost 66% to women’s 33%. And if 66% of the jobs lost were lost by men, then 66% of the jobs returning should be men’s jobs. But that’s easy in retrospect. Things like that are not always clear when you first come to a novel problem.

So, the important point is, as always, if you don’t understand something, plug some fake numbers in and play around a bit. For most problems like this it’s easy and inexpensive to do a thought experiment.

Are These Figures Possible? [Music Piracy Figures]

One of the simple types of experiments you can do is to “sanity check” a figure. The RIAA, a music industry lobbying group, claims that Americans consume up to $20 billion in pirated digital music a year.

So let’s see what that would look if every American was involved — in other words, what is the average amount that each American would need to pirate to make this figure possible?

Two important figures to keep in your head for sanity checking American stats — there are a bit more than 300 million people in the U.S. and about 200 million of those are between the ages of 15 and 65 (we call the 15-65 demographic the “adult” population).

Let’s take the 200 million figure and see how much pirated music each person would have to consume to reach a total of $20 billion each year:

• Assume an iTunes price of about $1 a song
• That means 20 billion pirated songs are being “consumed”
• 20 billion songs / 200 million people = 100 songs per person

(see http://www.wolframalpha.com/input/?i=20+billion+%2F+200+million if the zeroes get confusing for you).

• To reach the $20 billion mark, the equivalent of every person in America would have to download and listen to 100 different songs per year.

Obviously, this isn’t the case. A large percentage of 18-29 year-olds pirate something or other, but most don’t pirate much, and 30% of young adults pirate nothing at all.

Of course, usually these things come down to a lot of small-scale users, and some larger “superusers”. Though reality doesn’t fit nearly into groups and categories, we can imagine some discrete groups of “user types” to make our math easier.

For example, if we say that 50% of the adult population are small scale pirates who pirate an average of 20 songs a year (100 million * 20 or $2 billion), then we might imagine a class of “super-pirates” that make up 2% of the population and all of the difference (18 billion songs). With these parameters, how much pirated music do our super-pirates have to consume per year?

18 billion / (200 million * .02) = 4500

Well, maybe. People do get obsessed with downloading. But if someone downloads 4500 songs, does that really mean $4,500 was lost? Would 2% of the population — one person out of fifty — realistically pay $4,500 a year for the music if pirating it wasn’t an option?

And could those people really “consume” 4500 songs a year? Again – maybe. If you listened to a completely new album every day for 365 days a year, you could probably do it (although in that case you’re listening to each of these songs once). But is one out of fifty people doing that?  When those 50 people include not only your classmates, but your mom and her friends, your younger siblings, men and women? When they include people that don’t listen to music at all?

Of course, you can reconfigure these numbers various ways. Create more “super-pirates” or less, have the average person consume more, up the price per track to $1.50 or lower it to 65 cents, the portion of the price that goes to the record company instead of Apple. But the brief tour of this statistic seems to indicate that it is not very credible, although a lower statistic such as $2 billion might be possible.

None of this proves anything, of course. But with some very simple math, you can identify figures that are suspect, and either investigate them further, or just ignore them until more reliable data comes along.

What Would We Expect? [College Participation of Black Males]

A lot of figures in the media are presented as being shocking — they do call it “news” after all, and “news” tends to be synonymous with “unexpected”.

But when a shocking stat is presented to you, the appropriate reaction is to ask yourself what would be an expected stat. Very often, we react with shock to statistics without asking what would not shock us.

Take, for example, this headline, which made the circuit of Facebook shares in early 2012.

Four percent! That seems shocking. But how shocking is it?

Let’s assume that a non-discriminatory situation would be that black males were represented in college at the same rate that they were represented in general society. What would that rate be? Well, blacks represent about 12% of the population, and we can assume about half of blacks are male. So black males represent 6% of the general population. In a perfect, non-discriminatory world six percent of college students would be black males.

Was that the impression you got from the headline?

Keep in mind this is still not a good result — technically the difference between six and four percent means black males are under-represented by 33%. But it is in a different range than one might perceive the initial claim — and given the broad inequities of our primary and secondary education system, I’d be hard-pressed to say that this level of under-representation is surprising.

When presented with the unexpected, always ask what the expected value would be. When confronted with the unfair, always ask what the fair value would be. When confronted with the unprecedented, always ask what the historically expected value would be.

Resolving Contradictions: Survival and Mortality Rates

In some ways, this next one is an extension of the toy model idea, but for a more particular purpose.

Many times you will have what seem to be contradictory figures. You may hear, for example, that America has a much higher survival rate than England for prostate cancer, but that mortality rates of the disease in each country are about the same.

Before deciding that one of the statements must be a lie, it is always helpful to think through whether it is possible for both statements to be true.

In this case, we have to think through the difference in how survival rates are measured vs. how mortality is measured. When someone dies of prostate cancer, that counts as a mortality. The mortality rate of a disease equals the percentage of people in a given population who die from that disease.

What’s different about survival rates?

In short, survival rates are measured not only in relation to the percentage of people who die, but also to the number of people in a given population who are diagnosed. This distinction between survival rate and mortality rate is probably worth remembering.

When we encounter statistics, asking if the figures given are possible, investigating whether the shock-value of a statistic is valid, and resolving conflicting numbers are three ways to run good mental experiments.

This section deals primarily with the following elements of the COMPARABLE framework:

• L: Would the comparison benefit from a longitudinal/cross-sectional analysis?
• B: Were base rates considered? Were relative and absolute increases considered?
• P: Were the pictorial/graphical representations fair and unbiased? Did they help illustrate the data or were they eye-candy?

Note to instructors: We are still working out the language here. Both the terms “cross-sectional” and “longitudinal”, when used professionally, have more precise meanings than we are giving them here. For instance, under our definition, a case-control study would be cross-sectional, whereas that is not the way the word is used in practice. 

The bigger point is that we are less concerned about students knowing the precise names for study designs than we are about them understanding these two major ways of looking at comparison. You’ll note we’ve called these “approaches” to distinguish them from “study designs”. Maybe that’s enough. But if it makes more sense to make up names that don’t conflict with study design terms, we may do that, for instance, by calling Cross-sectional a “snapshot” approach  instead.

There are two types of comparisons you can make.  Or rather, there’s lots of different types of comparisons you can make, but we can split them largely into two groups:

• You can compare one thing (or person or place) to another thing (or a set of things, persons, or places).
• You can compare a thing (or person or place) to itself over time.

To get at why these two types of comparisons are different, consider a comparison of your school with Harvard. We test graduates of Keene State, and we test graduates of Harvard. The Harvard graduates score better on the test.

Does this mean the professors at Harvard are better? That the classes are better designed? That Harvard is more effective?

No, of course not. In this particular case we can’t be sure if the Harvard students were just smarter to begin with, or better at taking tests, or if Harvard was doing exceptionally well with its teaching.

What we are really interested in is the difference one thing makes (in this case, choice of school). One thing we might do is follow our golden rule and compare like to like. What if we compare the test scores of Harvard students that scored 1300-1400 combined on the SAT with Keene State students that scored the same? Since the SAT predicts a number of related variables, this is more of an apples-to-apples comparison.

Immediately, you see some problems though – are the students that get into Harvard with a 1300 SAT *really* the same as those that come to Keene State with a 1300?

An alternate way of looking at what Keene State does is to look at it longitudinally. If we give students a test coming in, we can give them the same test when they graduate , and look at how much better (or worse) they did. The nice thing about this sort of comparison is that we know we are comparing apples-to-apples because were are comparing the very same student before and after.

The first type of comparison we described (Keene State to Harvard) we will refer to as a cross-sectional analysis. The second type we call a longitudinal analysis. While these terms have specific meanings in the context of study design, here we will be using them more broadly, as follows:

• A cross-sectional analysis compares one thing to a similar other thing. For instance, we compare the test scores of Harvard students to Keene State students.
• A longitudinal analysis compares a thing to itself over time — for instance, the difference between pre-test scores of Keene State students and post-test scores.

It’s worth noting that many of the best comparisons involve both these attributes — For instance, we can take the longitudinal analysis a step further. We can run the same pre-test/post-test scenario at Harvard and at Keene. And then we can compare the increases/decreases on the tests for each student.

So if, on average, Keene State students gain 12% on the tests, and Harvard students gain 10%, we might say that Keene has more impact (again, as always, there are caveats around randomness and bias, but we’ll get to that later).

We like to get at problems using this last sort of approach when we can, but of course it takes a lot of effort. For the longitudinal comparison we describe there, we need to pre and post test. The test has to be the same at Harvard and Keene. And so on.

So while controlled longitudinal designs, on average, tend to be stronger study designs, they are not always possible. And sometimes a good cross-sectional comparison can get you better answers than a longitudinal comparison, assuming relevant factors are accounted for.

Therefore, the questions we want you to ask, when looking at a comparison, are as follows:

• Is the comparison cross-sectional or longitudinal?
• If it is cross-sectional, what might a longitudinal analysis tell us?
• If it is longitudinal, what might an alternate cross-sectional analysis tell us?

THE U.S. HEALTH SYSTEM FROM THREE DIRECTIONS

We can see how this matters in practice by returning to the health care example from Week One. The U.S. is often said to have poor health outcomes, especially considering how much money we spend on health care. The following chart shows how unusually poor our outcomes are given the money we spend.

[h/t Lane Kenworthy for this graph and approach to health outcomes analysis]

Let’s walk through this graph, which demonstrates a common way to look at cross-sectional approaches. On the Y-axis, you have life expectancy, which is considered a gold standard health outcome. On the X-axis you have per capita health expenditures.

What you want to be, ideally, as a country is up at the top left of this plot — high life expectancy and low cost. Japan manages to get that spot, living very long and spending very little, but most countries cluster towards the center here, spending about $3000-4000 per person each year with their citizens’ average life expectancy around 80.

The one big outlier here (and it really is a huge outlier) is the U.S., which spends an incredible amount of money on health care and has the lowest life expectancy of any of the comparator countries.

Now there’s a point to this graph, which is that the way the U.S. manages its health care system may be causing us to burn up all this money for no real health effect. The implication tends to be that we are the only well-to-do nation in the world that does not have a public health care system (and we still don’t, as of this writing, since the Affordable Care Act does not take effect for a couple of years).

Could the U.S.’s lack of a public health care system explain the stunning difference? Well — perhaps. But we have to go a step further. There are many ways in which the U.S. is different than these other nations. For instance, we are a fatter nation. We have a higher murder rate. We are a more rural nation, which makes quality health care harder to deliver. Any of these things are different enough that they might explain at least part of the pattern seen here.

So we go to longitudinal analysis. In the chart below, we show the scatter graph over time:

This graph shows a couple of things. First, even in 1970 we were somewhat inefficient, spending a lot on a mediocre outcome (compared to other nations at the time). But we weren’t *that* inefficient. As time has gone on, we have become more and more inefficient compared to other countries.

Now that we have the time based comparison, we can look at our factors again. We have a high murder rate, true, but it has fallen more in line with other countries over time, so that can’t be it. We are fatter, true — but actually, other countries have been catching up to us in that regard, so if that was the cause, we should see a narrowing of the gap. What’s more, there seems to be a very clear point around 1980 or so where our costs began to skyrocket and outcomes slowed — it is hard to think of any dramatic demographic shift that began at that time that could explain that.

Again, this is not to say it is definitely our health care system at fault. If anything, it adds a bit more nuance to that discussion, because we must ask in what ways our policy and economy in the late 70s and early 80s changed — we were never a great nation for health care, but we became truly horrible at it over a very specific period. We also have to realize there are delayed effects in health care — that drift you see after the 80s could be due to treatment not received in the 70s, or even 60s.

But longitudinal analysis allows us to eliminate some of the simpler answers, and gives more complexity to our policy question. In general, if presented with a longitudinal comparison, ask for a cross-sectional one. If presented with cross-sectional comparisons, ask for a longitudinal analysis. And if possible, get both at once, as we did here.