The horizon is not so far as we can see, but as far as we can imagine

The Curse of Knowledge and Predicting Electoral Results

I’m good at predicting some thing. Economic events, longer term political and social trends, and so on.

When I was younger, I was also good at predicting elections.

Then I started getting election results wrong.

It took me a while to figure out why.

I knew too much.

When I was young, I didn’t follow politics much, I just read the newspaper and maybe saw the occasional newscast.

So I knew what ordinary voters knew.

Later I studied economics and politics and so on. I started knowing the actual consequences of elections, and I knew what governments had actually done, not just what was reported enough to sink into someone who wasn’t paying much attention, like most of the population.

In other words, I was no longer a proxy for the population.

Nor does analysis which tries to model “what does the ordinary person know” really help.

(I am fundraising to determine how much I’ll write this year. If you value my writing and want more of it, please consider donating.)

The reason is simple enough: we don’t make most decisions, including voting decisions, rationally. We make them based on our feelings. The strongest feeling determines our vote.

Since my feelings, as someone who followed politics closely, were no longer in sync with those of the majority of the population, I could no longer use my feelings to judge. Even if I were to try and feel like someone different from me (not impossible) I still knew too much. My empathic sense of what they would want was based on more knowledge than they had.

And boom, lousy electoral predictions.

Talking with other close observers of politics and economics and so on I’ve noticed that this curse of knowledge is not mine alone. Most political experts predict elections badly.

So I’d keep thinking “wow, if you elect this dude it’ll be disastrous,” because I knew the consequences of what they were actually likely to do, and primary and general election voters would keep picking awful people.

The problem, of course, is that the consequences of actual government policies don’t go away. But the other problem is figuring out how to avoid the curse of knowledge: one needs it to make important predictions other than electorally, but if you don’t understand the electorate, that cripples any chance of good candidates being elected.

So far, I don’t know how to avoid this and if anybody else on the more left wing side does, I’m not seeing it.

A problem.



Homelessness and Poverty Are Policy Choices


China Already Has Almost 430 Million People With First World Incomes


  1. Well, you undoubtedly read too much as well. If you see something on the front page of the newspaper every day for two weeks, and repeated on the nightly news every night for the same two weeks, then you quite reasonably assume that virtually all voters know about it.

    The problem is that a fairly large percentage of voters do not know about it, and an even larger percentage don’t believe it because some blogger told them it wasn’t true. So, while you assume it will impact the election, it either doesn’t impact the election, or it has an impact other than the one you expect.

  2. Other than “read too much,” I meant to say that you assume that you “read too much into voters ability to read.”

  3. Crap. It’s early (Pacific time). One more time. You “read too much into what you assume that voters read.”

  4. Mallam

    I don’t think you’re right about this, or at least you’re going too far with it. Granted, it’s true to some extent. For example, you and I can look at Joe Biden and see him as a governing disaster, a political mediocrity who only had a political revival because he was chosen as VP and Hillary Clinton lost. However, “regular voters” opposed to Trump have all the reason in the world to at least consider him, and on a rational basis. Why? Because he meets all the generic criteria of a safe choice to a population that’s generally split 53-47. People like him personally, they they see polling confirming others also like him personally, and he was VP to a reasonably popular president. Matching that with “we need to pick who’s best to beat Trump”, and it makes sense. This is why he’s the number one choice among black voters. They don’t care that he backed policies against their interest in the past. It’s taken as a given that dude has contributed to racist state policies. But much of the underclass is very electorally pragmatic, they’re not stupid, and if anything they imagine who’s best to serve their political interests in the moment. And right now their interests are: who can drub Trump? Any other considerations are at best secondary. Further, he’s polling ahead of Trump in states like Texas, which signals he’d potentially have the best shot at actually governing because he’ll bring the Senate with him. “Accelerationists” should perhaps keep that in mind for next time (I’m not insinuating you’re one or advocate for it but it’s a pervasive idea on the left). If Kamala Harris wins Iowa or New Hampshire or South Carolina then all bets are off and the calculations change (which isn’t to say Harris is a good/bad choice either).

    Didn’t mean to get too specific, but I think it’s a good exercise to look at how regular voters are making rational decisions.

  5. bruce wilder

    I know lots of leftish sorts whose analysis of the electorate begins and ends with wishful thinking. For some of them, their thinking consists of imagining that “everyone” would respond favorably to a particular policy proposal or just a slogan. “Health care should be a right!” – – that kind of thing. Even professional politicians sometimes seem to have an oddly mechanistic view of how people will respond to particular cliches of rhetoric or symbolism.

    Most of the electorate does not know (or care) much about politics and does not have strong ideological commitments. Even some people who do seem to have “principles” seem to acquire them from comic books or Ayn Rand or some half-remembered high school history or Sunday school lesson.

    Moods and fashions and memes sweep thru the electorate because humans are monkeys and monkey see, monkey do: people are mimetic, we imitate the shiny objects that get our attention and being storytelling animals, we crave the meaning we get from stories.

    American politics is awash in manipulative propaganda and desperately short of institutions of verification and exemplars of good, humane, balanced judgment. There is therefore little check on elite manipulation of the electorate, other than the contest among rival elite factions. The competition among elite factions, as far as the propaganda it generates, generally repulses ordinary voters, makes them more ignorant and seeks to subvert whatever instincts they might have for good, humane judgment. “Good guys” are rarely on the ballot anyway and even when they are, the gargantuan effort to draw people into enough political participation to resolve their ambivalence, suspicion and ignorance into support is rarely made on the left.

    I think elections are eminently predictable for anthropologists and political scientists who can focus on observable indicators. The professional generation of propaganda has created technicians who are quite good at fine-grained prediction in service of control. Obama’s 2012 campaign against Romney was a masterpiece of tuning the propaganda to produce a narrow margin of nevertheless nearly certain victory.

    Even the intuitive readers of the zeitgeist did fairly well in anticipating Brexit and Trump.

    The “curse of knowledge” would seem to come down to knowing the consequences and wishing it were not so.

    For the left in general, the shortage of available resources and institutional base combine with an ideological blindspot regarding the need for same. I wish Tom Friedman would lose his spot on the NYT op-ed page for being wrong and a very bad hack of a writer and be replaced in prominence by someone wise and knowledgeable. if we are to have really good left writers policing our politics from the commanding heights of journalism they have to be able to eat and make careers being good and smart. there is very little space for that, for even Matt Taibbi or Glenn Greenwald.

    What can be said for pundits applies also to politicians. They have to be able to make careers doing good and being good. There is not much space for AOC and what little space there could be is closed deliberately by the powers-that-be. It is not accidental that Sanders comes from a small, rural state without overwhelming corporate interests.

    To get a better politics, we would need much better institutions to create an ecology for a better politics. That is predictable. (cf Hedges, Death of the Liberal Class)

  6. Dale

    Maybe it is not so much how much you read or watch as it is the source of your information that counts. Right through the last presidential election cycle the MSM were convinced HRC was the leader in the poles and the soon to be victor. But there were voices in the wilderness. Michael Moore for months warned everyone of how well Trump was doing in the rust best, and why. No one wanted to listen. Call it hubris, call it cognitive dissidence, call it whatever you want. Moore is today’s Cassandra uttering prophecies that none can accept as true. And there are others.

    My point is that no matter how many times the mainstream media say something it doesn’t necessarily make it true. Something like 85% of American news is controlled by five major international corporations. They are interested in profits not truth, whatever that is. Most people don’t even subscribe to newspapers anymore. They get their “news” from TV. Goebbels must be laughing out loud in hell. Most of us have been programmed to have a 20 second attention span. The diversity in opinions on any subjects still seems to be greatest in the books that are available to us. Getting these books into the hands and minds of enough people to make a difference is difficult to do. I have no idea how to change this.

    So Ian, it may not be important about how much knowledge you have. Maybe you are using the wrong sources for political projections. Ask the people in the street how they plan on voting, and why, will probably serve you better. These contacts have to be one on one, not via telephone or computer polls or what have you. If you ever do decide to do this, I would really be interested in what you find out.

  7. ponderer

    I think you have latched onto something important here. As I have argued when people have brought up “collective” responsibility for some misdeed, it takes a lot of effort and time to be informed on a topic. The people working 2 and 3 jobs have the least time and access to make informed decisions. All the more reason to treat our corrupt media with the disdain they deserve. The propaganda runs 24/7 but people know most of it is bullshit, they know they are being lied to even if they don’t know the truth of matters.
    I didn’t make my Trump prediction on knowledge, I did it on societal anger. The masses may be ill-informed but they are still angry at the state of the country. It doesn’t take a Oracle to see that the status quo is going to be at a disadvantaged. Now if I were asked about Biden vs. Harris I couldn’t say. The only thing I’m sure of is that both will lose to Trump.

  8. someofparts

    What can we do to destroy/replace this atrocious party?

    and this galling news too:

    If they just want Trump to win by a landslide this makes sense, otherwise, WTF?

  9. different clue

    @ Ian Welsh,

    The only way I can think of to self-mitigate the effects of the Curse of Knowledge is to accept the fact that it is a curse and that nothing will lift it or break it. One can then thinkyfeel about ways to work with it and/or work around it without forgetting about its ever-hovering presence.

    The only way I can think of to do THAT . . . when it comes to guessing election results . . . is to stay socially connected to many different types of bunchloads of ordinary people and keep monitoring what they say about things and stuff, and how they say it.

    If you think you have figured out a way how to do that, you can then make silent predictions to yourself. If you get it right over and over and over, this method is working and you can predict again.


    “Catfood Joe” makes Catfood-DemParty sense in the wider scope. The wider scope being that the Catfood Democrat Party has only one Prime Directive just now. And that is to never ever permit Sanders to get the Democratic nomination. And the SubPrime Directives 2,3 and 4 are to never ever let Warren or Gabbard or Gravel get the nomination either. Never. Ever.

    They may not actively “want” Trump to win by a Nixon vs. McGovern sized landslide. But they have already decided that a Nixon-sized Trump victory in 2020 is a price they will accept paying in order to make sure that no New Deal Revivalist ever gets DemParty nominated for President. Never. Ever.

    ” New Dealer keep on moving. This is a Catfood neighborhood.”
    ” God help you New Dealer if the sun sets on you here.”

    That’s the whole Catfood Democrat ethic in one dirty little Clintonite Obamazoid nutshell.

  10. Wheels coming off implies momentum, momentum implies we can make a pretty good guess as to where they will go. There was a whole series of books written in the past century based on the premise that when the population grows suitably large enough its behavioral patterns can be anticipated. I tried years ago to apply dBase – database technology – to human thought: when the database, the body of knowledge, grows sufficiently large enough so too our confidence in the intuitive leaps made during the conclusion drawing process. Or maybe it was something I read many (many) years ago in one of my rogue odd fellow grandfather’s books about druids. There can be glitches, deadends. But for the most part … traffic flaggers will tell you cars are like cows.

    What I have found of The Gift is you can’t use it. It’s one thing to know, quite another to even vocalize what you know. And it can be scarier than fuck. Ripples, on still waters. One word.

  11. Herman

    I used to have the problem that many on the left (and even some on the right) have which is I thought I knew the obvious best choices in elections and I felt that if someone could just put out a good platform and run a good campaign the rest of the population would see the light. Then I started to read up on politics and why it seemed to be so dysfunctional. Most of my ideas about democracy were shattered. Tribalism, identity, charisma and other factors outside of pure policy considerations were a lot more important than I thought.

    Also, I now realize that most people are just not very interested in politics and that it is not because they are stupid but because for most people following politics is probably a waste of time given the infinitesimal chance they have of actually having an impact on politics. Those of us who follow politics more than usual might even be more biased and more prone to ideological thinking than we realize and that is not even getting into how so many politics junkies treat politics like sports fandom.

    I won’t be surprised if Bernie loses to Biden or some other neoliberal Democrat. Even though I think Bernie is the best candidate out of the whole field, that doesn’t mean other people feel the same way.

  12. Dan

    I don’t see the problem. You can do both. You can understand how popular voter groups think and will be likely to vote, and you can understand the social, political, economic, and scientific realities that may have little or nothing to do with the voter sentiment.

    If this is hard to do, then someone is not getting out and listening or reading enough. You have to move around, talk to people, live in different places, and get inside the mentality of important subcultures. A lot of stuff is just universal and based in gender and family stuff, and a lot of the universal stuff gets mediated in predictable ways by particular ideologies and cultures you can study and learn to understand “from the inside.”

    The biggest gap in every society may be between its sincere supernaturalists and naturalists, but there is usually so much actual crossover between them, often such reaction and mutual mirroring, I am not sure there really is such a big divide in understanding as there is a refusal to grant each other any legitimacy. When these divisions are not politicized, there’s no problem and a comfortable, sincere non-understanding may set in because there’s no interaction happening.

  13. someofparts

    I only bother to follow politics in the first place to arm myself with foreknowledge as much as possible.

    Local news is on tv in the background as I write this. Just heard that Biden is in town. Big fundraiser in wealthiest part of the city. News lady telling us in hushed tones how much a seat at the dinner costs. Running a clip of him back-pedaling on the Hyde amendment faux pas. Stacey Abrams will be attending.

    I’m looking forward to watching Trump take him down.

  14. Hugh

    Looking at Presidential elections since Truman, I would say the biggest factor in predicting who wins isn’t policy. It’s the level of fatigue with the current Administration. This plays out usually in an eight year oscillation. A President has a couple of terms and people have had enough. His successor is too tied to the old Administration, and if the other party has anyone offering anything the least bit different, the other party wins. So in 2016, what the Democrats needed was to recast their message (Bernie Sanders), but instead they went with and even more status quo successor candidate (Hillary Clinton). Trump, on the other hand, offered something different. He won. I think there is already a lot of Trump fatigue and this will only grow. The Democrats could win with a simple call of a return to normalcy. But importantly, normalcy is not the same as the status quo (Biden). We want the ship of state to stay afloat. That’s normal. We do not want it to continue to sink, just more slowly than under Trump. That’s the status quo.

    It’s important too to realize that a new message doesn’t have to be real or even have that much content. Think Trump or Obama’s hopey-changey schtick. Most Establishment Democrats (like Nancy Pelosi) think the Trump fatigue is sufficient that they can run another status quo clone in the person of Joe Biden. They are making a fundamental mistake. There is as much to dislike in Joe Biden as there was with Hillary Clinton. And neither offer anything new (as opposed to Sanders, Warren). I would suggest that Sanders, Warren should incorporate much more in their messaging words like “new” as in a new way of looking at things, “different” as in a different way of doing things, “shake things up”, “restore trust”, etc. It is good to have policies to back this up but this is what people want to hear. It is what they are looking for.

  15. EverythingsJake

    Maybe running a check against the long list of Cognitive Biases would be, at least, a start?

    Wikipedia List of Cognitive Biases

    There are many sites that discuss how to avoid falling prey to Cognitive Biases. There’s a link to one I like below, it has good references to scholarly research. One thing I find is that just changing my aspect, mostly going to a different neighborhood shakes up my view. Additionally, I live in a non-homogenized area of the city, so I hear lots of viewpoints every day, from a large and growing contingent of homeless, and poor to middle class, even the recent invasion of the upper middle class provides additional perspective, fortunately or unfortunately, the wealthy class has not settled in here (yet).

    Wikipedia Avoiding Cognitive Biases

  16. Tom

    Ultimately society today is what Hypocrisy we want to live under and how many “others” we want to kill to make it possible.

    I should point out that French Occupation of Algeria caused women’s literacy to disappear as they closed down the schools for them, and women lost economic power as the Napoleonic Code stripped women of property rights. So the French and not Islam caused a retrograde of women’s rights and Muslim Men who grew up under the French System adopted the French Misogyny in their ideals on what Islam was about as Muslim Women no longer could read and point out they had numerous rights in the Quran that they were ignoring.

    It is also ironic that Islam and Catholic Christianity were pro-science which is often forgotten alongside the fact that the Father of Secular Enlightenment, Averroes, was a devout Muslim.

    We can’t even get our History right or place historical people in proper context. And to think we can do the same today?

    We’re screwed for another dark age and technological collapse to around 15th century tech levels at best.

  17. MojaveWolf

    I’m probably missing something, but unless you’re making bets about or investments based on who wins, then what’s the point of “predicting” anyway?

    I can see all sorts of reasons why one could predict that Bernie might not win or Tulsi probably won’t win (most coming down to effective propaganda by the DNC/MSM, some due to problematic campaign strategy on Bernie’s part, i.e. failing to tend his working class base, falling into the pundit class identity politics trap, fighting the battle on the wrong terrain in an effort to appeal to voters who won’t back you anyway while losing out on lots of others desperate for a champion who won’t lie to them like Trump or Obama), and probably too many variables to predict this far out anyway in any direction, but . . .

    unless a candidate wins who will actually turn around the course of the world (or someone so awful wins that they promptly start an exchange of nukes) . . . what good is predicting? Aren’t the more important things (1) figuring out (1) which candidates you think will make a sufficient, or at least might make a sufficient difference, for good, and (2) figuring out what factors will decide whether they can get elected, and how best to push those things in their favor? (along with: what traps not to fall into, to help ensure there is sufficient momentum to accomplish policy objectives) Those are where the predictions will matter (or, alternatively, figuring out how to counter the prevailing trends if your side loses, if you think they have no chance).

    Sometimes, also, if you think you have something worth fighting for, if all the models say “you have approaching zero chance”, well, as Sun Tzu said, if all else fails and doom appears upon you, “When on death ground, fight.”

    That’s what Bernie did in 2016, and barring cheating, it would have worked. I don’t think even he realized how much of anti-establishment groundswell there would be. The same thing worked for Trump and I don’t even think he was trying to win, at first (or even at the end, he didn’t want to lose, but he didn’t want the job, either, I’m near certain; his ideal would have been Hillary winning in a way where he could go on TV and make a jillion dollars talking about how he got cheated, but his reflexive combat instincts were sufficiently better than all the pros and HRC was sufficiently awful at campaigning that he won anyway, because unlike w/Bernie, the powers that be thought they had it locked up and there was no reason to cheat).

    If you’re a Tulsi or Gravel fan, that’s what to do now. Say what you mean, mean what you say, push the issues that are actually important, and come out swinging. The corporate media has improved their disinformation tactics and are trying to stomp out future Bernie uprisings before they can get out of low single digits, having realized past mistakes, but you never know . . .
    (and if you’re a Bernie fan, he’s still got at least as good a chance as anyone else) (and all the Dems have to worry about winning the primary w/out losing the independents; which the corporate media is making VERY difficult, one almost wonders if they want Trump to win, or if they’ve started believing their own false paradigm, or that much faith in their propaganda efforts and voter malleability–they might not be wrong on this one– or what)

    There’s reality, there’s the official version of reality, there’s what most people faced with both of these plus a few alternatives actually believe, plus conflicting priorities, etc. And this applies even to the people not paying attention–they think certain things are, and people are a certain way, and will vote accordingly. The propaganda artists have figured this out and are trying to convince voters Biden is a champion of the working class and “any blue will do, so vote for whoever our polls tell you is in first” and about 5,000 other bits of disinformation, but they don’t control public perception entirely and some people are getting better BS detectors, and most people don’t fit into the neat little boxes that most of the pundit class (including most of the leftist pundit class) keep trying to fit everyone into, so . . .

    All is flux. We need someone w/the sincerity, commitment and track record of Bernie, along w/the combat instincts of Trump, and ideally a lot of charisma. Don’t predict prevailing trends, try to make new and more favorable trends (as Bernie and Trump both did in 2016).

  18. MojaveWolf

    I seem to be setting a record for getting myself put in moderation for no discernible reason today . . .

    No doubt this utterly pointless test comment will go right through, tho!

  19. Ché Pasa

    The problem is that elections won’t — indeed can’t — get you what you want especially if you’re on the leftish side of the political/ideological spectrum. Electing candidate X rather than Y will inevitably disappoint and may well be horrible. Yet Y will inevitably be a disaster for some people in some ways some of the time, so it is better to go with X, no? At least you avoid that disaster. Or something.

    The only certain outcome is that somebody is going to get screwed, likely screwed badly as a consequence of this or that election, and there will be wars and horrors too many to count, and most of us will stumble on as best we can, and the cycle will repeat, over and over. Progress will be made in some areas, reversion in others. The imperial spectacle will increase and diminish; the narrative will be jiggered this way and that; decline and advancement will contend with stability.

    Predicting who will win the office is more of a media marketing ploy in the end. Wanting to know and understand the future is human nature, and media makes plenty of money asking its questions and predicting its outcomes. Wrong or right, it doesn’t matter. The prediction is the product.

    Our nihilists yearn for The End, Finality, the Inevitable Collapse and see the signs everywhere. Well, sure. An end is certain, at least on the individual level if not necessarily collectively all at once (though when the Giant Asteroid comes… ) Yet stumbling on, one way or another, is just as inevitable. Elections merely change some of the players at the top, but they’re all socialized to the same system and inevitably perpetuate it. Even some outlier like Trump (who’s not really an outlier at all, but more an unmasked representative of his class) serves and perpetuates the System in spite of the spectacle, just with a somewhat different cast of characters.

    So it goes.

  20. NRG

    You, Ian, make a valid point, and one that apparently has broad application. Take for instance, the IARPA prediction tournament, which asked people to make predictions about particular future events or trends. Experts in fields relevant to the question consistently did worse than broadly-read amateurs. The results have implications for all sorts of topics, with electoral politics surely being one example. A story on predictions by experts, and the tournament, bearing directly on your thesis:

  21. Watt4Bob

    There is one factual understanding that is relatively new to most of us, and has vastly complicated our efforts to predict political weather, and that is the growing awareness that the democratic party is not really opposed to the corporatist agenda that is pushed by the GOP.

    It may as well be yesterday that we realized that the democrats are republican lite.

    This realization has lead rather quickly to the realization that 10% of the population, regardless of political identification, the credentialed class, does not give wit about the rest of us, so long as they can hold on to what they’ve got.

    One of the main, unforeseen developments is the obvious commitment of the 10% to ‘pulling up the ladder’, that has those who we previously assumed, on thin evidence, were ‘fighting for us’, instead, joining the effort to impose austerity.

    When one considers how blind to this obvious reality the die-hard HRC backers of a certain stripe are, then we can see why we have such a hard time predicting.

    We had been aggressively asking why poor working class whites were voting against their own self interest, all the while ignoring the glaring spectacle of poor ‘liberals’ blindly voting for the latest Clintonite, third-way to the exact same effect.

    And now, finally, we’re starting to understand that identity politics is not evidence of the democrats dedication to a ‘big tent’, but a deliberate, and proactive tactic to divide and prevent an effective collaboration toward solidarity.

    All of this, though you and I have been mulling it over for ten years or so, may as well have started dawning on ‘us’ just yesterday.

  22. different clue


    Some people understood this long ago and tried explaining it to the rest of us. Several decades ago Kurt Vonnegut wrote an essay which is in his book of collected essays called Palm Sunday. The essay is titled In A Manner Which Must Shame God Himself.

    Its been many years since I read it. So I can only paraquote the following quote which I won’t be able to remember exactly. Vonnegut was writing about a social gathering of some of the Literary and Political Beautiful People. . . . people like Barbara Walters and Kenneth Galbraith and other famous names. He noted that among themselves they considered that ” whether someone was a Democrat or a Republican was considered a hilarious accident which no one was expected to explain.”

Powered by WordPress & Theme by Anders Norén