Why I hate standardized tests

Image by Peggy Monahan. Thanks Peggy!

One time my friend Bev asked me to help her with the GRE. I told her that I hate standardized tests and assured her that she didn't really want my help. She insisted that I was the only person she knew with math skills so I went over to help.

First question: If one car mechanic can fix a problem in 2 hours and another one can do it in 3 hours how long if the work together?

My head practically exploded at the absurdity of this question. First of all, if two car mechanics try to work together they'll end up drinking, smoking, and bull-shitting and nothing will get done. In the unlikely event that they actually tried to work on the same car at the same time it wouldn't get done faster because car repair is probably one of the least parallelizable tasks imaginable. I mean, what are they going to do -- both pull on a wrench at the same time and extract a nut twice as fast?

Of all the tasks in the world they could have chosen -- painting a wall, canvassing a neighborhood, etc -- they pick a nearly worst-case example. That said, understanding serializable vs. parallelizable tasks is extremely valuable knowledge so I spent 20 minutes explaining pipelining and caching strategies and then Bev understandably fired me exactly as predicted.

Here's another example from a Wonderlic sample test:


I simply abhore this questions and "puzzles" like this one -- it's completely subjective. I can make a case for all 5 of these being unique. 1 is the only one who's longest diagonal is equal to sqrt(2) of its sides. 2 is the only one that can be created by moving a single vertex from a rectangle. 3 is the only one with an anspect ratio greater than two. 4 is the only one that has regular angles greater than 90 and is also the only one with 6 sides. 5 is the only one with 2 acute angles. How is it that number of sides is somehow more important than the other features?

Questions like the two above make me feel that the author is a moron and that fact immediately makes me angry: Where does this moron get off judging me? And that gets me to why I hate standardized testing. It's a game about guessing what the author wants you to say using rules of thumb and pre-described algorithms versus demonstrating that you are capable of independent thought. And in a world full of computers that will slavishly follow endless and complicated pre-describred tasks, we don't need humans to do the same.

I don't merely reject standardized testing as a means of judging people's abilities, I reject the premise that standardized testing demonstrate anything positive -- society does not need more people who excel at slavishly following rules of thumb and formulas as those people's jobs are soon to be replaced by computers if they haven't been already. We need people who understand, who create, who invent now more than ever and this simply is not tested by standardized tests. While it is certainly the case that there are people who do well on standardized tests who are also creative, it is even more so the case that there are people who are very creative who nevertheless fail at standardized tests and unfortunately the tests tell those people "you suck" instead of "you're awesome".

Thinking like a chemist

Photo from flickr user zcreem

Suppose you were asked to mass produce a toy train car -- four little wheels connected to a small wooden chasis. Being an inhabitant of the 21st century you'd know exactly what to do: build an assembly line. After some tinkering you'd end up with some sort of jig that held the body into place and attached the wheels in a nice predictable fashion so that it could be repeated rapidly. Henry Ford would be proud of you.

But what if the wheels of the car were as small as atoms and the chassis a single molecule? Then how would you do it?

Such molecular assembly has been done for a long time, it's called chemistry. But chemists don't think about the assembly problem the way Henry Ford did. Their approach is fundamentally different.

A chemist would build the molecular toy car by putting four bazillion copies of the wheels into a bag full of water with a bazllion copies of the chassis and then they would shake the bag intensely assuring that some fraction, perhaps a minuscule portion of the original, would assemble themselves *by accident* into the toy car.

A chemist would then come up with a way to separate the fully assembled cars from all the left over parts that failed to assemble -- perhaps the vast majority of the pieces. For example, the chemist might dump the contents of the bag out over a pine tree. The un-assembled pieces, being smaller than the fully assembled pieces, might fall through the branches more easily while the the larger assembled pieces would be more likely to snag a branch and get stuck on the way down. As a result, the top branches of the tree would be more likely to hold the desired product. Then a chemist would snip off the top half of the tree and shake out the contents and repeat the process, knowing that for each repetition they have purified the sample a little bit. Having never even laid eyes on the target, they would declare that their end product was, say, 99% pure.

Seems absurd? It is. But it's also incredibly clever -- it permits the manipulation of atomic scale things by exploiting the fact that you are much, much larger than they are and can therefore easily move around quintillions of molecules inside of a "bag" no bigger than a drop of water. As a filter they'd use something like gelatin which at the molecular scale is a lot like the pine tree -- a big furry mess of interconnecting obstacles that would let some things pass though easily while inhibiting the movement of other things. As hard as it is to believe, this technique is accurate enough to allow separation by incredibly tiny differences in mass. Below is a picture of an actual gel, the light pink parts are the molecular batches (in this case DNA) that have been pulled through the gel which is the purple background. You can make out six tracks in this gel which is six different runs.


photo from wikicommons Gel_electrophoresis_2.jpg

Below is a picture of someone loading a gel. You can make out 10 "lanes" they are loading for their experiment. Into each lane they are placing a tiny drop of water using a pipette which is just a fancy eye-dropper. Each drop they are inserting into the gel might contain billions upon billions of the molecule of interest.


Photo from flikr user rocksee

Understanding Principal Component Analysis via cool Gapminder graphs

Gapminder.org is a wonderful site full of "statistical porn". This chart in particular is a fascinating graph that demonstrates the correlation between income and child mortality rates. It is also a great example to teach about a cool statistical tool: "Principal Component Analysis".

In this graph of regions there is an obvious negative correlation between infant mortality and income illustrated by the fact that the data points scatter along a line from upper left to lower right. In other words, if you knew only the infant mortality rate or the income of a region you could make a reasonable guess at the other.

Principal Component Analysis (PCA) is a statistical tool that’s very useful in situations like this. PCA delivers a new set of axes that are well aligned to correlated data like this -- I've illustrated them here with black and red lines. For each axis, it also returns a “variance strength” which I’ve represented as the length of the black and red axes. (Actually I just hand approximated these axes by eye for the purposes of illustration).

The strongest new axis returned by PCA (the black one) aligns well with the primary axis of the data. In other words, if one were forced to summarize a region with a single number it would be best to do so with the position along this black axis. The zero point on the axis is arbitrary but is usually positioned in the center of the data (the mean). Positive valued points along this black axis would be those regions further toward the lower right and negative valued regions would be those further toward the upper left. Let’s call this new axis “wealth” to separate it in our minds from “income” which is the horizontal axis of the original data set. Increases in “wealth” represent an increase in income and drop in infant mortality simultaneously.

The second axis returned by PCA is shown as the red axis. Countries that lie far off the main diagonal trend-line (black axis) have particularly unique infant mortality rates given their wealth which we’ll assume is because of something unique about their health care systems. Points well below the black axis are regions that have very good health care given their wealth and those above it have particularly poor health care given their wealth.

Because PCA gives us convenient axes that are well aligned to the data, it makes senses to just rotate the graph to align to these new axes as illustrated here. Nothing has changed here, we've simply made the graph easier to read.



Before you even look at specific regions on these new axes, one could guess that socialist countries would score more negatively along this red axis and those whose economy is heavily biased towards mineral extraction -- where income tends to be very unevenly distributed -- would score more positively. Indeed, this is confirmed. The most obvious outliers below the black axis are Cuba and Vietnam where communist governments have directed the economy to spend disproportionately on health care and the outliers on the other side are: Saudi Arabia, South Africa, and Botswana -- all regions heavily dependent on resource extraction where the mean income statistics hide the reality that few are doing very well while the vast majority are in extreme relative poverty.

One particularly interesting outlier is Washington DC which is located as far along the red axis as is Botswana! In other words, based on this realigned graph, you might guess that the wealth in DC is as unevenly distributed as it is in Botswana. Fascinating! (The observation is probably at least partially explained by the fact that it is the only all urban "state" and urban areas will tend to have wider income distributions than rural/suburban areas.) Also note that all of the points in the United States (orange) are well into positive territory on the red axis -- our health care system is as messed up relative to our wealth as is Chad, Bhutan, and Kazakhstan -- countries with completely screwed-up governmental agendas. Think of it this way: the degree to which our infant mortality rates are "good" owes everything to our wealth and is despite the variables independent of wealth! In other words, countries that provide average health-care relative to their wealth like El Salvador, Ukraine, Australia and the UK fall right on the black axis but we fall significantly above that line -- roughly the same place as countries that are, independent of their wealth, really messed up like Chad and Kazakhstan. (A caveat: the chart is on a log scale so the comparative analysis is more subtle than I'm making it out here.)

PCA returns not only the direction of the new axes but also the variance of the data along those axes. To understand this, imagine for a moment that all the regions of the world had exactly the same health care given their income; in this case all the points would align perfectly along the main trend line (the black axis) and the variance of the red axis would be zero. In this imaginary case, the data would be “one dimensional”, that is income and infant mortality would be one in the same statement; if you knew one, you'd know the other exactly. Now imagine the opposite scenario. Imagine that there was no relationship at all between income and infant mortality; in that case we would see a scattering of points all over the place and there wouldn’t any obvious trend lines. Neither of these imaginary scenarios are what we see in the actual data. It isn’t quite a line along the black axis but neither is it a buckshot scattering of points, so we can say the data is somewhere between 1 dimensional and 2 dimensional. If both variances are large and equal to each other, then the system is 2 dimensional while if one of the variances is large while the other is near zero, then we know the system is nearly 1 dimensional. In other words, PCA permits you to summarize complicated data by finding axes of low variance and simply eliminate them. This technique is called “dimensional reduction” and is a very powerful tool for summarizing complicated data sets such as would arise if we looked at more than two variables. For example, we might include: car ownership, water accessibility, education, average adult height, etc to the analysis at which point performing a dimensional reduction would help to get our heads around any simplifications we might wish to make.

Macro-scale examples of chemical principles

I like macro-scale examples of chemical principles. Here's two I've noticed recently.


I was very slowly pouring popcorn into a pot with a little bit of oil. The kernels did not distribute themselves randomly but instead formed some long chain aggregations because, apparently, the oil made them more likely to stick to each other than to stand alone. This kind of aggregation occurs frequently at the molecular scale when some molecule has an affinity for itself.


This is wheelbarrow chromatography. During a rain, water and leaves fell into this wheelbarrow. Notice that the leaves and the stems separated; apparently the stems are lighter than water and the leaves are heavier. This sort of "phase separation" trick is frequently used by chemists to isolate one type of molecule from another in a complex mixture. Sometimes the gradient of separation might be variable density as in this example, but other times it might be hydrophobicity or affinity to an antibody or many other types of clever chemical separations known generically as "chromatography". Note that the stems clustered. Like the popcorn above, apparently there is some inter-stem cohesion force that results in aggregation as occurs in many chemical solutions.

Mathematical pedagogy II

One time I overheard a mother say about her daughter, "My girl is great at math except she has a hard time with word problems." I thought to myself was: "Then your girl isn't good at math!" Only in school is "math" a game of manipulating disconnected numbers. Math is word problems (although they become increasingly abstract). As many people do, this mother was confusing arithmetic for math.

Arithmetic is not the same things as math; the distinction is very important. Mathematics -- the exploration of pattern by means of proof -- has not substantially changed in the last 2500 years. But arithmetic -- the computing of numerical values -- has changed beyond all recognition in the last 50. The fastest human arithmetical savant might be capable of adding a few numbers per second. To say computers eclipse that isn't even close. Even a cheap cell phone can do 100 million arithmetical operations per second; an inexpensive computer can do 3 billion per second. Going from 10 to 10 billion in 50 years isn't "change", it's "conquest". It is absolutely absurd to ask a human being to do an arithmetical task. When your free cell phone, cheap watch, or low-end TV can out-perform you by a factor of a billion, it's time to stop competing.

An analogy. For thousands of years people have fetched well water by gracefully carrying large jugs on their heads. One day a pipeline is built and water can be delivered directly into everyone's house. The pipelines didn't improve the porting of water, they eliminated it. Imagine a teacher saying to a class: "We must learn the art of porting water on our heads! Carrying water on your head is good for your balance, strength, and poise!" When inevitably the children don't want to practice because they can just get water out of a tap, the teacher says: "The children of today are all loafers! Pipes are making everyone stupid and lazy."

Teaching arithmetic in 2009 is like making pupils port water on theirs heads. Arithmetic is a conquered technology. But for one exception -- estimation, to which I will return -- arithmetic simply isn't done by hand anymore. Game over.

Yet "math" in grades 2-6 is still dedicated to arithmetic. (We'd be better off teaching them porting water on their heads, at least that would promote exercise!) As I suggested in an earlier blog, I'm afraid that the principal accomplishment of existing numerical pedagogy is to make a significant portion of the population hate numbers. To illustrate: despite the years dedicated to the subject in school, you can't find any cashier nor any restaurant waitstaff who can sum a bill sans computer, much less compute tax, despite the obvious service advantages. Nobody can or wants to do arithmetic anymore; it's time we admit this and move on. Yet there seems to be a strong reluctance to change the curriculum because of some society-wide nostalgia for arithmetic.

In my opinion, eliminating arithmetic from grade school curriculum is a fantastic opportunity -- now we can dedicate those precious grade school years not to boring arithmetic but to fun and useful math!

In an age of ubiquitous computers lack of even rudimentary computer skills abound. The required skills are more than functional knowledge of email and web-browser -- what's missing is the knowledge of how to frame questions so that they can answered by a computer. The first tool that should be learned is the spreadsheet. If 50 years ago it was necessary that all people should be able to add and subtract, today it is necessary that all people should be able to use a spreadsheet.

Here's what I would do if it were my own children I was teaching. As would anyone, I would start with concepts of magnitude. How big is 7? Make a pile of 7 blocks. Is 9 a bigger than 3? A lot bigger or a little? Next, in continuity with existing curriculum, I'd introduce the concepts of addition and subtraction by adding or removing to a pile. But after that I'd break with tradition. From here forward I'd have my kid sitting in front of a computer spreadsheet program like Excel. We would explore questions learning how to program the spreadsheet to get an answer. We start with questions like "If I had had 3 blocks and then I was then given 6 by Ann and 4 from Bob, how many total would I have?" For this problem, we'd put "Mine" in a column and 3 next to it, then we'd put Anne and Bob's contributions in the next rows and finally type in the excitingly magical phrase "=sum(B1:B3)" and, presto, there's the answer. I'd advance through problems like this always with the of practice solving word problems by converting them into spreadsheets programs. There would never be a big sheet of arithmetic that the pupils was supposed to mechanically work though as is currently the case in grades 1-4. All problems would be word problems. All answers would be spreadsheets.

As we advanced from calculation with spreadsheets into algebra, we'd return to a somewhat traditional curriculum of symbol manipulation but using the computer to sanity check our conclusions and to solve associated word problems. But there would be a few changes I'd make.

First, as before, I'd never present a page full of meaningless algebraic equations to "solve for x". That kind of practice is too mechanical for my tastes. That said, I do think that algebraic manipulation is useful and has to be practiced, but I'd keep it grounded with word problems in every exercise. Second, I'd work to make sure the students can use algebra as a language -- that they can translate questions to and from that language. A game I've played with students before is what I call "reverse word problems". Given an equation like: "5 = x + 3" I'd say: "I have 5 oranges now. I had 3 and my friend gave me some number I can't remember, we'll call it x. How many did he give me?". Once students get the hang of this game, I have been amazed how quickly they can connect these meaningless abstract symbols to their lives. Once when playing this game a girl told me a used-car-lot-based story that involved a sale price on a used car. I loved it and asked her how she came up with it and she told me that her father sells used cars; this was a girl that just 30 minutes earlier had told me that she hated math because she didn't see the point of it!

A common response to proposals for introducing calculators into math curriculum is that the students who rely on calculators don't have any intuition for numbers and are prone to draw ridiculous conclusions when they mistype something on the calculator. For example they are capable of writing "10+5 = 50" because they pressed multiply instead of add and have no intuition for that being absurd. This is a very valid concern. I would address this in two ways. First, I would never use a calculator in a classroom, only spreadsheets. Calculators show only the current operand and never show the operation so the user can not go back and check that things were entered correctly. Calculators are therefore so error-prone that I never use them except for extraordinary circumstances. Second, and more importantly, I'd teach number estimation as completely separate exercise from computation. Estimation is wildly undervalued and should be taught regardless of whether or not you believe in eliminating the arithmetical curriculum. (As an interesting aside, I've noted that all my nerd friends over 60 can estimate extremely well because they were all slide rule masters which required them to keep track of magnitude in their heads while using the slide rule for the mantissa.)

I would teach number estimation with just two key ideas. First is the idea of an order-of-magnitude. Strip off all the significant digits and just deal with the magnitude. Multiply and divide are then just slapping on and off zeros. I'd do all this exercise with classic Fermi problems such as estimating the volume of the earth and so forth. All of these exercises would be played (with supervision) with Google. Google the diameter of the earth. Simplify this to an order of magnitude, cube it, etc. Then write down your order-of-magnitude estimate and then look up the real answer with Google. The second tool I teach is doubling and halving. Want 8 times of something? Double it three times. Want 9 times? Double it three times and add some more. Want to divide by 5? Half it twice and take some off.

In summary, if teaching my own children math, I'd abandon arithmetic, emphasize spreadsheets, avoid calculators, always use word problems, ground algebra with reverse word problems, and teach estimation as it's own computer-less skill with Fermi problems and Google.

How tools become abstractions, The Value of tinkering, and Mathematical pedagogy

I started programming computers when I was about 10; I had no idea what I was doing of course, I just followed the examples in the Apple and TRS-80 books. Meanwhile, part of tinkering with a computer also meant using the stock tools such as the commands to copy files, list directories, or fetch the time. I don't remember the exact day it happened, but there was some point that I realized that the commands like "copy" and "dir" were just programs like the ones I was writing -- that I was capable of making, in principal if not yet in practice, the tools I was using. That realization is a kind of magic moment in mastering a medium -- the moment when you see a tool beyond its immediate utility into the deeper concept that it embodies. You go from a tool-user (imitating the tool's use with variation) to a tool-maker (exploiting the under laying principal).

Tinkering begets insight. For example, when you tinker with woodworking tools you use the tools, hammers, drills, nails, etc. for their intended purpose without considering how they work. A nail is a device which accomplishes the task of attaching two boards, and a first you don't see past that utility. But as you become more intimate with the process at some point it may occur to you that a nail is just a kind of friction joint or that a screw is just a wedge wrapped around in a circle or that a drill is a kind of special scraper. When you see these kinds of things your mind opens up. If a nail is a friction joint then what other kinds of friction joints are possible? You might invent a dowel joint having never seen one before. If a screw is a curved wedge that buries itself into the material, what other ways can a wedge be integrated into a structure?

There are many people who never pass the tool-using stage of their craft. They stay within the existing rules of their medium and are perfectly comfortable there. I don't mean to be critical of that approach. That said, I can't help but think that the reason that some craftsman don't go past the tool-using stage is that for whatever reason they simply haven't had enough "a-ha" moments to internalize the drive to abstraction. One reason for this, I believe, is that most formal pedagogical practices try to shortcut the process of tinkering. There's a strong temptation when teaching something to cut to the chase; many teachers end up acting as if teaching someone something is about giving them answers instead of guiding them to the answer.

While I'm advocating the advantages of coming to understand tools as abstractions, I don't think that this learning process can be far removed from tinkering. As proof, I submit the typical pedagogy of mathematics as an extreme demonstration of how abstraction without tinkering goes horribly awry. To illustrate, imagine that you took a shop class but had never before seen any woodworking -- nails, screws, drills, all of it was totally new to you. Imagine if the teacher began the first day of class by saying: "OK class, this is a hammer. A hammer is characterized by a relatively large hard mass attached to short lever usually, but not always, made of wood. Note the counter-balanced curved metal head called a claw. Everyone pick up your hammer now and follow along with me banging the hammer on the table. One (BANG), Two (BANG), Three (BANG). OK class, now this is a drill. A drill is a helical scraper whereby a pitched screw... blah, blah, blah." Imagine this kind of boring technical analysis of woodworking tools day after day. On the last day of class a pupil raises her hand and asks: "Teacher, I have paid close attention and have learned about all of these tools, but I'm not clear on what it is you DO with these tools." and the teacher responds: "I'm not really either sure as I've never done it myself, but I think people make cabinets and chairs and stuff like that, more importantly all of this *is* on the college entrance exams." The kids think: "What?! You mean this class was about making chairs?!"

While this parable is exaggerated, I think that mathematical pedagogy (and other subjects) is not far from this. Most teachers and students intuit addition and subtraction and after that it's a free-fall into meaningless technical abstraction. By third grade, multiplication and long division become so bogged down in symbol manipulation that almost no child leaves understanding what multiplication and division *are*. Later they'll use them as a tool to perform yet even more technical abstractions but all too often 12 years can go by and the pupil still has no idea what you *do* with these tools other than make more abstract tools. I am convinced that the primary societal accomplishment of current mathematical teaching is to make people hate math. Those few nerds who make it past the abstractions were going to do so even if they hadn't been for the classwork, those who were going to hate it just hate it more, but the majority who might have liked it / used it / appreciated it instead come to despise it after being bludgeoned by it for 12 years. So the net result is that we may be worse off teaching math classes than had we simply done nothing. (My opinions on how to reform this are best left for another day.)

Even in the best case where an excellent and enthusiastic teacher is well practiced and accomplished in the use of the relevant tools of their art, the teaching techniques tend to lack tinkering because of its "out of control" sense. But, without discovering for yourself that a hammer is a solution to a problem, you can't easily appreciate the tool's function and you finally can't abstract the principal of the tool to other problems. If you tinker enough, and if you are lucky enough to have access to someone who has crossed the abstraction bridge, then you can grow from tool-user to tool-maker and, to me, that's the difference between ordinary works and extraordinary works.