There was a time when education was a politician’s refuge. When a candidate wanted to look caring and populist, they would go to a local school, take some photos, and proclaim that children are the future (preferably sung along with Whitney Houston) and the US (or fill in your constituency) has the best schools in the world.
Then along came No Child Left Behind and TIMSS and a desire to quantify success. Three important cultural dynamics came to play: 1) the idea that value can be concretely demonstrated, 2) a strong cultural desire to be better than everyone else, and 3) a lack of time and/or capacity to assimilate the enormous amount of information available. To satisfy these three elements, the public needs a number that tells us our kids are learning and our schools are the best. Recently, many policy makers have also decided we need a number that will tell us if teachers are any good.
All of the ways this doesn’t work is a book, not a blog. The enormity of variables makes saying anything substantial nearly impossible. Either you make assertions based on statistics that ignore factors (lurking variables is the term in the research world), or you make the scope so finite that you have said nothing.
To that end, I want to ride on Dan Meyer’s coattails and create an educational policy version of pseudocontext and call it pseudo-data analysis. I define PDA as any attempt to use statistics to draw a conclusion that is either a) drawing or allowing oversimplified conclusions to be drawn from data, or b) trumpeting one piece of data while ignoring the glaring gaps.
1) A climatology professor in Washington state who has determined that math textbooks are to blame for his students failing a self-created diagnostic test.
2) The headline: “Texas’ white eighth-grade students earn second highest score in nation on science NAEP”
This would be a great context to ask students to conjecture on how Texas is in the top ten in every subpopulation, but 26th overall.
Every teacher has seen PDA that is annoying in the least. Please share any other examples you can find.