I first came across this study by George Mason University's Daniel Klein and Zogby International's Zeljka Buturovic, which appeared as a journal article in Econ Journal Watch, which Klein edits, in a link at Tyler Cowen's site several weeks ago. Cowen links to about a dozen interesting pieces every day, and I thought Klein's study was so obviously flawed that it wasn't really worth commenting on. But now it has re-appeared in the editorial pages of the Wall Street Journal, with the somewhat non-sequitur title, "Are You Smarter Than a Fifth Grader?".Here's what Klein and Buturovic did. They took a survey using one of Zogby's internet panels, which is by far the worst polling instrument that they could have selected. The panel was not weighted and was not in balance. For example, McCain led Obama 49-43 among respondents to the survey, even though roughly the opposite outcome was observed in the actual election -- and only about 4 percent of the respondents were Hispanic and only 39 percent were female. Then they asked 16 "questions of basic economics", as the Journal's sub-head describes them, and arbitrarily included eight of them in their analysis but threw the other eight out.If you were expecting fifth grade questions about supply and demand, you'd be wrong. Let me just say: I come at this as a University of Chicago economics graduate who indeed disagrees with the liberal orthodoxy on many economic matters. But questions such as "[Does] poverty cause crime?", which was one of the questions that Klein and Buturovic excluded without explanation, are more like Zen meditations than matters of basic economics.Others were poorly phrased, for instance: "A company that has the largest market share is a monopoly?". This is confusing; having the largest market share is a necessary, though hardly sufficient, condition for being a monopoly, and no alternative definitions were presented. Is this question really an objective basis for determining whether someone is more or less "enlightened" (that's actually the term that Klein uses!) about economics?Some come closer to having a technically correct answer, but are more within the realm of trivia. For instance, "In the USA, more often than not, rich people were born rich?". Notwithstanding that the definition of "rich" is ambiguous, you could probably develop an empirical answer for this question based around studies of social mobility. But unless you'd spent a great deal of time reading the academic literature on mobility -- something that few laypeople will do -- there's really no way that you'd know it.Finally, there are some questions about which there is considerable disagreement even within circles of academic economists -- as Klein should know, since he's commissioned several surveys of them. Economists are about evenly split, for instance, when it comes to the minimum wage. There is much closer to being a consensus on free trade, but there are an ample number of heterodox views. And in other cases -- like the Rand Paulian view that "More often than not, employers who discriminate in employee hiring will be punished by the market?" -- there is very little in the way of recent academic research at all.So basically, what you're left with a number of questions in which people respond out of their ideological reference points because the questions are ambiguous, substanceless, or confusing. Klein is blaming the victims, as it were.There would have been much better ways to construct a study like this one. For instance, questions could have been developed from standardized tests of high school students, like the AP Economics exam, or from surveys of academic economists. Such studies might well support Klein's thesis. But between the poorly-considered questions and the poor choice of survey partner, this amounts to junk science.
WSJ editorial page = G.I.G.O.
A Wall Street Journal op-ed citing misleading/ambiguous/tendentious claims? I am shocked. Shocked.
Maybe there should be a requirement that all such surveys include "WTF?" as a possible answer to each question.
Ha!. Just like that "bias survey" someone posted last week. Instead of asking factual questions they ask more "Zen-based" interpretative questions to measure knowledge. Pretty fun to see how researchers/pollsters game their questions to get a desired response. Kudos to Nate for uncovering alot of this.
Wasn't there a GMU econ prof with whom there was a previous tiff?
My first email addy: garbageingarbageout@btw, back in the day may have totally underestimated the amount of BS on the net, if possible, as nowadays it's a cottage industry w/Reps ...or as Bush would say misunderestimated the amt. of misinformation, song and dance, deflecting nonsense ~ conservative winger trolls can spew at liberal blogs!that is all
I seem to remember a time when the WSJ editorial page was a bastion of intelligent conservatives who used logic and data to back up their beliefs.What happened to that era? Even the few honest intelligent conservatives left--David Brooks for example--are shunned by a large part of the movement.
Nate let the authors off easy imho on how they cherry-picked only 8 Q's from among the original 16 [actually 21] poorly-worded/phrased Q's as polled by Zogby.here is the author's own words to describe their 'decision-making':from page 3 of 23:'The Eight Questions We Use to GaugeEconomic Enlightenment'"Of the 16 questions in that uniform format, this paper deals with only eight.We have omitted 8 of the economic questions in that format because they are not as useful in gauging economic enlightenment, either because the question is too vague or too narrowly factual, or because the enlightened answer is too uncertain orarguable."sad, but true... the old saying that stats don't lie, but liars use stats [to prove anything they want to prove]wv - phuman [faux humanoid]
@Jacob: Rupert Murdoch bought the WSJ.To answer the question in the headline, "yes".To the Zen-oid question "Does poverty cause crime?" I offer the following Zen-oid answer:The poor you will always have with you, but you will not always have me.
these are the 8 Zogby Q's that the authors used to determine 'econ enlightenment':The statements of the eight questions used are the following:1. Restrictions on housing development make housing less affordable.• Unenlightened: Disagree2. Mandatory licensing of professional services increases the prices of thoseservices.• Unenlightened: Disagree3. Overall, the standard of living is higher today than it was 30 years ago.• Unenlightened: Disagree4. Rent control leads to housing shortages.• Unenlightened: Disagree5. A company with the largest market share is a monopoly.• Unenlightened: Agree6. Third-world workers working for American companies overseas are beingexploited.• Unenlightened: Agree7. Free trade leads to unemployment.• Unenlightened: Agree8. Minimum wage laws raise unemployment.• Unenlightened: DisagreeWOW - every progressive's nightmare scenario... and how subjective [you had to agree or disagree]Minimum wage, free trade, out-sourcing jobs overseas, gov regulations, monopolies, housing, and 'standard of living'surprised they did not plant Universal Health Care and socialism in there !!!this would be a poor study if conducted at the HS level in an ESL setting...
Why would anyone think we would need to conduct a survey to determine economic knowledge by political leanings. The very fact the left actually believes "New Deal" fascism ended the Great Depression and creating the world's largest Ponzzi Scheme was an intelligent way to address retirement security, tells you everything you need to know about the economic literacy on the left.Oh well, I guess: "In the long run, we will all be dead"
Thank you, Nate, SO MUCH for posting this. I've been seething about this idiotic editorial all day (and the underlying study since I first read about it on either Yglesias or Ezra Klein's blog, I forget which).Beyond what you already pointed out:- It would be very easy to choose a sample of economics questions with answers that are agreed on by most economists that could be biased in either a liberal or a conservative direction, since laypeople don't know as much about economics as economists and will go with their gut. If you ask about countercyclical spending, conservatives will be more "economically unenlightened". If you ask whether increasing the tax rate will ALWAYS increase revenues, liberals will look more "economically unenlightened." You not only would need to take questions from some standard econ source, but you would need to select those questions randomly and in an unbiased way. It is horribly obvious that the authors of this questionnaire did no such thing.- As you point out, this is a professor at George Mason University, a relatively well-respected school. It's disgusting that an academic would cite the results of a poll that's this obviously atrocious. Your post title speaks to this, but it's worth emphasizing.- Klein writes that the results of this poll should lead you to believe that "many of our leaders and their constituents are economically unenlightened". That's also an obvious fallacy. If you want to know whether liberal LEADERS are economically unenglightened, you would need to do a poll of the leaders. I'm not sure polling the population at large could really tell you about the economic knowledge of the leaders who they vote for. The general population is just not that knowledgeable about specific policy issues, whereas legislators (to some extent, probably not the extent you'd hope for) have to be.
Michael AndersonAgreed--we need such a survey, if for no other reason than to determine whether any conservatives who use terms like "fascism" or "ponzi scheme" understand what they actually mean.Though I suppose the answer to that is pretty self-evident.
Re: Michael. Exactly! It's just like how the left doesn't believe that the severe economic meltdown wasn't caused by the poor, poor banks and mortgage companies being cruelly forced to offer mortgages to poor people. Darn you poor people! If only you hadn't wanted houses and the banks weren't forced to reap ridiculous profits from you. When will the country learn?
Not to mention that very few if any on the left would contend that far right policies brought us out of the great depression.
Oh, one more point -You should also note that participants are asked about the 2008 presidential vote and their party affiliation before they're asked the "economic questions", priming them to answer the economic questions in a partisan fashion.
Some of those questions are worded in ways I would change. I'll agree to that. However, several of them are perfectly fine.For example, Nate brings up the "monopoly" question:"This is confusing; having the largest market share is a necessary, though hardly sufficient, condition for being a monopoly, and no alternative definitions were presented."This doesn't matter. Disagree is still the correct answer if the definition is not fully met. All companies with the largest market shares are not monopolies, so quite obviously the correct answer is disagree.It is a similar case the question regarding Free-trade leading to unemployment. Free-trade in and of itself, does not cause unemployment, so disagree is again correct. You could say he's baiting libs here, but again, I don't think you can contest the answer. Same thing with the exploitation question.
2. Mandatory licensing of professional services increases the prices of thoseservices.• Unenlightened: DisagreeAs a business owner in a professional service that began mandatory licensing about 15 years ago (I've been in the business for nearly 25), I can personally attest to the fact that fees are actually the same, not accounting for inflation, today as they were prior to mandatory licensing. My fees and hourly rates are exactly the same numerically today as those with my experience and quality were 15 years ago at the inception of licensing, and I'm now on the high end of the scale. 3. Overall, the standard of living is higher today than it was 30 years ago.• Unenlightened: Disagree define "standard of living." I'm sure I'm not the only one that feels far more stressed today than I did 30 years ago. And in our idealized fantasies about the "good old days" our quality of life seems lower today, regardless of political affiliation. Bogus questions. Only one of those 8 has a "correct" answer.
Nifty that I email 538, requesting an analysis of this poll and it is posted within minutes. I am truly impressed by your responsiveness ;]I think you make good points about the flaws in the questions and methodology.
@WallyThat's quite a reach. If it's a yes/no question, then it's also wrong to say that the company with the largest market share is not a monopoly, as it may well be. And of course if a company is a monopoly, then it must have the largest market share. That is intentionally confusing wording.And it's an even greater reach to suggest that it's "incorrect" to say that free trade causes unemployment. Free trade leads to factors that drive up unemployment. Or were you disputing the point because it creates jobs in other places? One could easily take "free trade causes unemployment" to mean that "free trade leads to higher unemployment rates."
Michael Anderson said... Oh well, I guess: "In the long run, we will all be dead"~~~~~~~~~~So many bitter, disingenuous, misinformation 538 conservative trolls who are woe is me, sore losers re: being in the minority after those years of prosperity/competence 2001 to 2008!so little time ...>btw, when are winger trolls gonna stop whining!rhetorical question
I'm sure the WSJ will admit to its despicable incompetence, right?Wait, why are you laughing?
were we to accept that the questions are remotely close to adequate (I wouldn't), as a paper, this wouldn't pass peer review in any halfway decent journal for the simple fact that omitted those other eight questions without also running an analysis including them and reporting it, at least, in a footnote is poor form.re: wally. Yes, I'd say he was baiting liberals into giving an "incorrect" answer because they've been primed to be consistent with their liberal views (no to free trade). Their own table shows that effect--the "non-partisan" questions (which I'd argue might have separate problems, but it's not worth describing the omitted variable of occupation here) show much more parity in "INCORRECT" answers.The idea that the questions on free trade, for example is "fairly neutral with respect to partisan politics" is laughable.
I saw this article earlier today and my first thought was: When is Nate Silver going to prove how much of a juvenile and inaccurate SHAM this study was?!?!?! First they use Zogby, add in some of the most confusing statements ever, and then prove the bias of the study by claiming those who answered "wrong" as "unenlightened." Bravo!
Nate,Your regular evisceration of bad statistics & slanted studies is inspiring, not to be too aggrandizing, but I mean it. You make it seem so easy, a multiple regression as casual as a trip to the bathroom. That your criticism was directed at a paper that was published in an academic (albeit not-too-noteworthy) journal gives me more than a passing glance of perspective in my own academic work. Thanks.& on that note, what is the state of economic journalism if the WSJ publishes a professor's op-ed with such dubious science backing it? People can make glib comments about Murdoch / Wall St. / etc, but remember that very smart, successful people run that paper & read it daily. We should never cease in asking fundamental questions about our information.Been a fan of yours since early 2008. Keep up the important work that you do so well.
Hey Nate,How 'bout some new senate rankings? Looking forward to seeing more blue towards the top.
the only half-way interesting result the authors came up with was the correlation they 'determined' between arbitrary 'enlightened' correct answers and respondent self-described ideologies:from pgs 10-11 of 23:'Lack of Enlightenment by Self-Identified Ideology'A question about ideology ran as follows: [7]Which description best represents your political ideology?• Progressive/very liberal• Liberal• Moderate• Conservative• Very conservative• Libertarian• Not sure• (Refuse to answer)Table 2. 'INCORRECT by Ideology (using twopoint scale for question responses).'Progressive(n=577) 5.26 (SE=0.07)Liberal(n=742) 4.69 (SE=0.06)Moderate (n=1086) 3.67 (SE=0.06)Conservative (n=1423) 1.67 (SE=0.04)Very Conservative (n=540) 1.30 (SE=0.06)Libertarian (n=369) 1.38 (SE=0.09)TOTAL [average]* (n=4737) 2.98 (SE=0.03) =====================================yes, they arrived at a direct ideological corelation [BIG surprise] - so the authors claim this supports their subjective determination that the Progressive/liberal responses are 'unenlightened'...WTF - but this is perfect for the WSJ and/or the Economist
Jacob,"If it's a yes/no question, then it's also wrong to say that the company with the largest market share is not a monopoly, as it may well be. And of course if a company is a monopoly, then it must have the largest market share. That is intentionally confusing wording."Um, no, it isn't. Make a venn diagram of all companies with the largest market share and companies with a monopoly, do the companies with the largest market share all reside inside the monopoly group? If not, the answer is clearly disagree. And if this question about monopolies, which is obviously a simple question of understanding the definition, was so confusing, why did so many more liberals get it wrong? Can conservatives and libertarians just think there way out of a box a little better? Is that your point?"And it's an even greater reach to suggest that it's "incorrect" to say that free trade causes unemployment. Free trade leads to factors that drive up unemployment. Or were you disputing the point because it creates jobs in other places? "That's one thing yes. But largely I was referring to the factors that drive unemployment are so much more complex than free-trade, that the only reasonable answer is disagree.
@Mike: I wouldn't make it too easy for the respondents just to choose WTF. I would give them a choice between WTF and Pie.
JURISthey needed to offer them 'lifelines' too or whatever Jeff Foxworthy uses on the 5th grade Q&A'ya know ya r a redneck when ya luv ya dat monopoly'
@Wally said: That's one thing yes. But largely I was referring to the factors that drive unemployment are so much more complex than free-trade, that the only reasonable answer is disagree.Which means that the only reasonable answer to every question above is disagree. Why did so many conservatives get so many questions wrong?
While I'm certain that there is a number at which a minimum wage increase would cause inflation, I've never heard of an actual minimum wage increase even being appropriately blamed for one actual inflationary period. Of course that never stopped the reactionaries from crowing like it's the end of capitalism each and every time.Can anyone show any study based on real numbers which could show even the weakest coloration? Perhaps we can put that old talking point.
@WallyFree Trade does cause some unemployment. Ideally, it will be temporary and those people will be redeployed more productively. If so, then it should reduce unemployment in the long run, but it certainly causes some unemployment during a "transitional" stage. Choosing to ignore that transitional stage entirely is -- at best -- a political decision, rather than a matter of economic enlightenment.Similarly, question 6 can be easily read as a true statement of fact -- there are third world workers being exploited on behalf of American companies. (And note that it didn't say "all".)Given the poor wording and the political priming, this test could probably be more accurately described as a test of economic political views than as a test of economic knowledge.
Jim,Why do we assume we're even talking about going from non-free trade to free trade?The question just said free trade causes unemployment. So, the simplest interpretation is to compare employment of a free trade system to non-free trade system. Not a transition in the same industry.
While I'm certain that there is a number at which a minimum wage increase would cause inflation, I've never heard of an actual minimum wage increase even being appropriately blamed for one actual inflationary period. Of course that never stopped the reactionaries from crowing like it's the end of capitalism each and every time.Can anyone show any study based on real numbers which could show even the weakest coloration? Perhaps we can put that old talking point.First of all, it's pretty much a proven fact that liberals are morons when it comes to basic economics. Now, about minimum wage...I don't think the argument (from conservatives) is so much that increasing the minimum wage causes inflation, but rather a gross distortion of the market wage rate for certain jobs. This distortion of the market wage rate for certain jobs leads to a misallocation of labor resources and is ultimately a net negative on the economy. Granted, it's a very small "negative" but it does exist. Now, maybe you liberals think that guaranteeing SOME (nowhere near MOST and certainly not ALL) people that work a "decent living wage" is worth the net negative that guaranteeing that wage is to the economy - and we can agree to disagree on that - but you cannot deny that having a minimum wage DOES have negative impacts and DOES distort the efficient allocation of labor resources in the economy.
@WallyWhat else could it mean besides transition towards "freer" trade? I don't think any reasonable person would interpret that question to mean that "nations with fewer trade restrictions must automatically have higher unemployment than those without."People interpret question meanings in the context of the real world. In this case, someone might reasonably interpret the question as asking whether free trade leads to more people being unemployed in their country, in which case the "enlightened" response is wrong.
@WallyThe questions about free trade and unemployment requires many studies to determine its effects and then is open to interpretation and how one looks at it.A much better question would be about comparative advantage, which covers most of the concepts behind free trade without getting into the elements which are open to interpretation.
Clever deflection Muley. Yes, a minimum wage increase always causes (often negligible) inflation, but the question concerned unemployment.
A minimum wage will always increase unemployment. The question is: does it affect you, or does it only affect young black males? If it only affects young black males, then most liberals will be in favor of it.
@shiloh I completely agree with you that the quote by John Maynard Keynes: "In the long run, we will all be dead" is both bitter and disingenuous, as well as morally bankrupt.@Domino . Had the left sounded the alarm bells prior to the housing collapse I might take them seriously. The only ones I remember hearing sounding the alarm bells prior to the housing crisis were those of the libertarian side who warned about artificial low interest rates and moral hazard created by Fannie and freddi.@Jacob I am not using the term Fascism as a slur as it is used in modern parlance but as a description of its original manifestation. the National Industrial Recovery Act, passed in June 1933, which set up the National Recovery Administration. Most industries were forced into cartels. Codes that regulated prices and terms of sale transformed much of the American economy. The industrial and agricultural life of the country was to be organized by government into vast farm and industrial cartels. This was corporatism, the essence of fascism.
At the very least, several of those questions are so badly written that answering them as the idiot wanted would be literally and provably incorrect."Third-world workers working for American companies overseas are beingexploited." Very literally, if at least two third-world workers employed by American companies are being exploited, then this is specifically and provably true. Furthermore, even if you step back from that literalism, it is still true that anyone who works for someone else is being 'exploited' - despite the negative connotations, it really does mean something closer to 'utilized for gain.'"Free trade leads to unemployment." Similarly, if there is at least one person who has become unemployed due to an expansion of free trade, this is an indisputably literally true statement. Further, even most right-wing economists acknowledge that some workers will lose their jobs as a result of free trade policies (even if they then retrain and move to new jobs, they still became unemployed). There is no way to answer this as 'disagree' without being baldly ideological. And I actually do support free trade to the fullest practical extent, but this is just plain a 'Do you believe Republican orthodoxy' question, not a factual one.
prog/liberals averaged about 5 'wrong' answersconservatives averaged about 1.5 'wrong/ answersmoderates averaged 3.67 'wrong' answersthe overall average was 3 'wrong' answers...conclusion = they 'proved' that the USA is a center-LEFT country since the MODS were closer to the LIBS than CONS...LOL - fun with statsthis 'study' fails basic quantitative analysis
There's no doubt that there are several flaws in this study, many of which are acknowledged in the actual paper, and it could have been done better.But 3 of the 4 questions listed here are ones that were EXCLUDED.Nate, you're bashing them for excluding some of the questions and while you criticize them for the quality of the questions excluded!You're right that they should have given reasons for the exclusions, and that there are flaws with the questions (included and excluded), but you should mention that it's possible that they excluded the questions that you're criticizing for the exact reasons you list. The study could have been done more professionally and honestly, but so could your critique.
In case no one is noticing, Michael Anderson is bitch-slapping this thread into submission.
Its fine for you guys to argue the fine points of the specific questions. But I think you are letting the trolls distract the conversation, which is their goal after all.The more serious problem with this study is the selection of the topics. Economics is rife with conclusions that are disliked by one side or the other. If the researcher picks the right questions, the outcome is guaranteed.This study is junk. Maybe libs are less informed about economics on average, or maybe they aren't. This study does not actually address the question.
Michael Anderson said... Had the left sounded the alarm bells prior to the housing collapse I might take them seriously.~~~~~~~~~~So basically, after (8) years of cheney/bush you can't take anyone seriously, ehas putting words in conservative trolls mouths is easy also ...>again, when are conservative trolls like MA gonna stop whining!take care, blessings
Read Full Article »