VOTAMATIC

Today, Civiqs is announcing the release of its public opinion Results Dashboard: an interactive website with data, charts, and tables from Civiqs’ daily polling of political attitudes all across America.

Civiqs is a polling and data analytics firm that conducts public opinion research online. Since 2013, Civiqs has fielded over half a million scientific research surveys and collected more than seven million responses to survey questions. (More background on Civiqs, and its relationship to Kos Media, is in this post from Markos Moulitsas. Civiqs has a separate team and is a distinct firm from Daily Kos but is owned by the same parent company.)

For the first time, Civiqs is making the results of its polls open to the public. The results dashboard is updated daily with current measures of President Trump’s job approval rating, the 2018 generic House ballot, favorable ratings of the Democratic and Republican parties, attitudes towards gun control, and much more. All of the results can be filtered and tabulated by age, race, gender, education, and party identification.

I’m the Director and Chief Scientist at Civiqs, and I’m extremely excited to bring this data to the public. Before founding Civiqs, I was a Professor of Political Science at Emory University; and before that, a pollster with firms in Washington, DC, and California. My academic research was largely focused on improving the measurement of public opinion, especially as it trended over time, and on forecasting attitudes into the future. Civiqs applies and builds on that research.

So much of the political debate these days is about how events and actions will play out in public opinion. And there’s not a lot of reliable polling available to answer that question. Often we find ourselves dependent on sporadic survey releases with inconclusive results, or polls put out by interested parties, and reporting that cherry-picks the largest outliers to tell misleading stories about what’s really happening.

The Civiqs research methodology represents what we believe is a better way to track public opinion on the political issues that matter most: with daily polls that detect shifts in attitudes immediately; and methods of data analysis that give us reliable, granular insight into the views of key demographic and partisan subgroups.

Civiqs polls on a wide range of topics, all across the United States, all the time. With results updated daily, the Results Dashboard can reveal shifts in public opinion both large and small, at the topline level and within small subgroups, as they happen. We optimized the Civiqs trendline results model to identify when changes occur—but to not overreact to short-lived, random deviations in the data. Oftentimes, public opinion remaining stable is just as interesting as when it changes—especially when events that seem important at the time have no impact on public perceptions whatsoever.

Here’s an example. Civiqs has been tracking attitudes towards gun control for over three years, starting in January 2015. At that time, Americans were evenly divided over support for stricter gun control laws. Shootings in Charleston in June 2015 and San Bernardino in December 2015 did little to affect public opinion.

But by the Pulse Nightclub shooting in Orlando in June 2016, Civiqs found that attitudes had begun to shift. The immediate reaction to the Orlando shooting was an increase in support for gun control legislation of about 3 percent. The Las Vegas shooting in October 2017 pushed support for gun control even higher. And after last month’s mass shooting at Stoneman Douglas High School in Parkland, Florida, support for stricter gun control measures surged by 8 percentage points more.

Using the “Refine By” options on the results page, you can filter the results even more finely and see that most of the increases in support for gun control after the Parkland shooting occurred among Independents and Republicans—especially those with higher levels of education. Remarkably, since January 2015, Republican women with postgraduate education have shifted from being opposed to stronger gun control laws by a 65 percent margin to being 3 points in favor of those laws, directly following the shooting.

How does Civiqs polling work? Every day, Civiqs surveys thousands of Americans about their opinions on politics, news, and current affairs. The respondents are sampled scientifically from Civiqs’ online, opt-in survey panel. To select panelists for interviewing, Civiqs draws a representative random sample of individuals from a registered voter file. Those people are matched to demographically and geographically similar individuals in the Civiqs panel. To achieve an accurate representation of the population, panel members from groups who are underrepresented in the Civiqs panel are sampled more frequently, and those from groups who are overrepresented in the panel are sampled less frequently.

Selected panelists are notified by email and complete the surveys in a web browser or smartphone at civiqs.com. Civiqs aggregates the results and applies a specialized statistical procedure that calculates trendlines, and models the data to be representative of the underlying population. (Technically, it is a Bayesian dynamic linear model with poststratification.) The results are published at civiqs.com/results, and updated daily.

The Civiqs survey panel represents all regions of the United States, all different groups, and every position on the ideological spectrum. Nobody is left out. The Civiqs panel is open to everyone who wants to join—Democrats, Republicans, Independents—everyone. The opinions of every single survey participant—on every side of every issue—are important.

Civiqs is focused above all on accuracy. The survey methodology implemented by Civiqs’ software is scalable, repeatable, and fully automated. Online polling methods have been studied extensively and been found to generate accurate results when researchers follow a series of best practices in sampling, user experience, interview design, and statistical modeling. Every Civiqs survey is conducted and analyzed in exactly the same way, which eliminates many of the errors and biases that arise in traditional polls. Response rates to Civiqs polls are more than double the industry average. Most importantly, Civiqs never does any ad-hoc weighting or adjustment of its results on a question-by-question or day-to-day basis.

I invite you to join in! When you sign up to take polls with Civiqs, you will receive periodic emails—about once every few weeks—when a new poll is ready. Most surveys are shorter than eight questions and take less than a minute to answer. You can even answer a poll quickly on your smartphone. Part of what makes Civiqs polls accurate is how easy they are to complete! Of course, you are welcome to unsubscribe at any time.

Civiqs will update the Results Dashboard each day with the latest survey results. Some questions will provide complete breakdowns of the results by demographic and geographic subgroups, in both the charts and crosstabs. Other questions will display the topline results only, with more information available to Dashboard subscribers. I hope you find the Dashboard to be a useful, informative, and detailed way to follow public opinion.

Cross-posted at Daily Kos Elections

Last week, just before Election Day, we published our final presidential election forecast: Hillary Clinton, 323 electoral votes; Donald Trump, 215. As I wrote when making that prediction, “While it’s possible for Trump to defy the polls and win the election, it is not likely. Our model estimates Trump’s chances at around 12 percent.”

Trump’s Election Day victory, with 306 electoral votes, took us by surprise. My forecast was wrong. It’s time to understand why.

The forecast was based on a statistical model that analyzed nearly 1,400 state-level pre-election public opinion polls, in combination with a set of political and economic “fundamentals” that added information about the election’s historical context. The fundamental factors (which turned out to predict the national vote share very closely) indicated that Clinton faced an uphill climb from the very beginning. In May, I estimated that Clinton’s baseline probability of victory was around 35 percent.

But all summer long, right up to Election Day, the polls told a different story. Pollsters reported that Clinton was consistently ahead, both nationally and in the states, by what were sometimes very large margins. By July, Clinton’s advantage in the polls lifted her chance of winning to 65 percent, and it never fell below that mark. After the first presidential debate, Clinton’s lead over Trump in the state polls was so great that our model gave her a 96 percent chance of victory. And our reading of the polls was not unique: Every other major forecaster also expected Clinton to win (albeit with varying degrees of certainty). It would have taken either a major campaign event, or a major failure of public opinion measurement, for her to lose.

The polling failure was what we got. Late campaign developments like the Comey letter may have affected some voters, but if so, polls still never showed Trump in the lead. In previous elections, the error in the aggregates of the polls typically went both ways, sometimes benefiting the Democrat, and other times benefiting the Republican. This year, the errors were massive, and they almost all went in the direction of Trump.

State-level presidential polls—especially in the swing states—were badly and systematically wrong, by amounts not seen in decades. The polling averages indicated that Clinton would win Florida and North Carolina by 2 percentage points, Pennsylvania and Wisconsin by 5 percentage points, and Michigan by 7 percentage points. Instead, Trump won all five, for a total haul of 90 electoral votes. The state polls were so inaccurate that Trump almost won New Hampshire, where he’d been trailing by 5, and Minnesota, where he’d trailed by 9. Across all states, on average, Trump’s margin of victory was 5 percentage points greater than our polling aggregates expected it to be.

Given this data, no reasonable poll-based presidential forecasting model could have predicted a Trump victory. There was no interpretation of the available public polling data that supported that conclusion. This was not a case of confirmation bias or analysts reading into the data conclusions that they wanted to see. The evidence supporting a Trump victory did not exist.

The miss was not confined to the public polls, which are often considered to be of lower quality than the proprietary research commissioned by parties and campaigns, and never released to the public. Reports suggest that neither the Clinton nor the Trump campaign saw this result coming. Neither did the RNC. Going into Election Day, Trump campaign analysts calculated that they had at best a 30 percent chance of winning.

Some forecasting models did give Donald Trump a higher probability of winning; most notably the FiveThirtyEight model at 29 percent. But the reason why they saw Trump’s chances as being more likely was not because they had a fundamentally more pro-Trump interpretation of the data. Rather, they put less trust in the polls, which increased their uncertainty in the overall outcome of the election in both directions. This widened the range of potential electoral vote outcomes seen as consistent with the data—resulting in their forecast of Clinton’s chance of winning getting pulled back towards 50 percent. No matter the level of uncertainty in the final outcome, every poll-based model’s best guess was that Clinton would win the same set of states totaling 323 electoral votes, and every model was wrong in the same way.

It is not yet known why polls underestimated Trump’s vote share so badly. The polls also overestimated Clinton’s vote share, but not by nearly as much. Survey researchers are already busy investigating different theories. One clue, however, was that there was an unusually large number of survey respondents, all year, who said that they were either undecided or supporting a third-party candidate for president. I mentioned this pattern in my final forecast, and you can see it illustrated in the chart below:

When as many as 12 percent of voters are uncommitted going into Election Day, it makes a big difference if they “break” disproportionately towards one candidate or the other. Nobody knows if there were significant numbers of so-called ”shy” Trump supporters who were uncomfortable telling pollsters they were backing Trump in this uncommitted bloc. But evidence from the exit polls suggests that many Trump voters “broke late,” and decided to support him only at the very last minute. Allowing for this possibility is something that should have contributed more uncertainty to most forecasters’ projections, including our own.

I checked whether the forecasts might have been wrong because one or two polling firms reported especially inaccurate results. That wasn’t the problem. In our database of state-level presidential polls, the two largest contributors were SurveyMonkey and UPI/CVoter, which together accounted for 29 percent of all polls. In many states, half or more of our data came from one of those two firms. I removed all of those polls from the dataset and re-ran the model. The results did not change in any meaningful way.

That so many people were caught off-guard by the election outcome suggests that the polling failure was a symptom of a deeper, industry-wide problem. Survey research is currently struggling through a difficult period of technological change. Fewer people than ever are willing to respond to polls, and those that do respond tend to be non-representative; older and more white than the population as a whole. Differential partisan non-response—in which the partisanship of people agreeing to take polls varies by their level of excitement in the campaign—causes poll results to swing wildly even if opinion is stable. This year, more polls than ever were conducted online, but the quality of online methodologies differs greatly across firms.

Despite these challenges, many media organizations and polling firms chose to undertake the hard work of surveying voters and releasing their results to the public, for free. There isn’t anyone who doesn’t wish the data had been more accurate. But those organizations who made the effort to contribute to public knowledge about the campaign by publishing their results deserve our gratitude and respect. Thank you. What we need in order to avoid a repeat of this surprising outcome in 2020 is not less pre-election polling, but more—by more firms, with different methodologies, and different viewpoints.

Cross-posted at Daily Kos Elections

Over the course of this presidential campaign, Daily Kos Elections has logged 1,371 state-level presidential polls into our database. All signs point to a Hillary Clinton victory.

Our forecasting model indicates that Clinton is highly likely to win key states including Colorado, Pennsylvania, New Hampshire, Virginia, and Wisconsin. In all five of these states, Clinton has never trailed in our average of the polls—and if she carries all of them, she would win the election over Donald Trump with 273 electoral votes, three more than the 270 required for victory. In addition, our model also favors Clinton in Florida, North Carolina, and Nevada. Together, those states contribute another 50 electoral votes.

That gives us our final prediction: Clinton 323 electoral votes, Trump 215.

Given that the forecast is based almost entirely on public polling data, how much can we trust the accuracy of the polls? As recently as one week ago, Clinton held such a commanding lead that our model placed her chances of winning as high as 96 percent. Since then, the race has tightened, and we currently estimate Clinton’s odds of victory at 88 percent. That’s enough of a change that a large and consistent polling error could make the difference for Trump. But the error would have to be very large, and very consistent. Going into Election Day, Clinton’s average lead in the polls is 3 points in New Hampshire, 4 points in Colorado and Pennsylvania, and 5 points in Wisconsin and Virginia.

Polling is never perfect, but systematic errors across multiple states in the same presidential election are historically not that large, or that common. Instead, the state-level errors form a distribution: In some states, one candidate outperforms the polls, and in other states, the other candidate does better. For example, in 2012, on average, the polls underestimated Obama’s vote share by a small amount; nevertheless, in 22 states, his polling was higher than his eventual vote share. Polling errors are less “correlated” across states than you might expect.

What about the magnitude of the state-level polling errors? Aggregating public polls usually produces forecasts that are very close to the actual outcome, especially in competitive states where pollsters have conducted larger numbers of polls. Again using 2012 as an example, there were 15 close states where a candidate won by 10 points or fewer (counting only the major-party vote). In seven of those states, the polls accurately predicted the margin of victory to within 1 percentage point. In another three states, the polls missed the actual margin of victory by under 2 points, and in four states, the polls were off by between 2 and 3 points. In only one state did the polls miss the margin by more than 3 points. And to reinforce our point above about correlated polling errors, Obama outperformed his polls in eight of the 15 close states; in the other seven states, Romney did better than expected.

So, while it’s possible for Trump to defy the polls and win the election, it is not likely. Our model estimates Trump’s chances at around 12 percent.

Stepping away from the polling data, there are reasons to think that the probability of a Trump victory isn’t even this high. None of these other factors are formally built into our model, and I haven’t analyzed them in any systematic or historical context, but consider everything below here informed conjecture. My Daily Kos colleague Stephen Wolf also examined some of these factors, and others, in a recent post exploring why the polls could be off.

First, our forecasting model takes the public polls essentially at face value: We apply a slight adjustment to polls conducted by partisan pollsters, and we make a few assumptions about how quickly to assimilate new polling data and how much to infer state trends from national trends. But we have no way to account for phenomena like differential partisan nonresponse, which may be responsible for the seemingly large swings in the presidential polls this year. If, contrary to some of the raw polling data, public opinion has been as stable as recent research suggests, then some of the more sophisticated online tracking surveys, like those from YouGov, NBC/SurveyMonkey, and Google—all of which have shown Clinton with a consistent lead—might have it right.

Our model also does not incorporate data on early voting, beyond what is implicitly captured by polls that include respondents who have already voted. Although there is disagreement about how much should be read into early vote totals, one state stands out: Nevada. Heavy Latino turnout in the Nevada early voting period appears to have put a significant dent in Trump’s chances of winning there—a must-win state for him where polls alone suggest he has at least a one-in-three chance of winning.

Related to this, there are a variety of reports indicating a large discrepancy in the size and quality of the Clinton and Trump campaigns’ voter turnout operations. In short, Clinton enjoys a significant advantage. Research suggests that her superior “ground game” could be worth up to 1 to 3 percent of the vote. This will not be picked up in the polls.

Finally, although the “fundamentals” of the presidential election have long been factored out of our forecast in favor of newer polling data, two key structural factors have actually gotten better for Clinton over the course of the campaign: President Obama’s job approval rating is on an upswing, and the national economy is growing at a faster rate than when we first accounted for these factors back in June.

There is one last caveat that gives me pause: The number of voters telling pollsters that they are still undecided, or are intending to vote for a third-party candidate, remains unusually high. We know that these respondents are disproportionately younger, white voters who would otherwise be likely to support Hillary Clinton. But we have no way of knowing for sure how these individuals will vote, or if they will turn out to vote at all. It’s something that I will be looking out for on Tuesday.

Happy Election Day.

Please head over to Daily Kos Elections, where you’ll find the implementation of my forecasting model this year. My forecasts are also part of the comparison table at The Upshot at the New York Times, labeled ‘DK’.

In addition to predictions of the presidential race, we are making forecasts of all of the 2016 Senate races, and we’re calculating a complete set of poll-tracking trendlines for every state. Here’s the North Carolina presidential matchup.

For polling trends in all 50 states at a glance, you can also check out the poll tracker page at this site, or my trend detail page, which gives a zoomed-in look at how voter preferences have shifted during the campaign.

Cross-posted at Daily Kos Elections

Even before news broke this weekend about Donald Trump’s 2005 Access Hollywood tapes, he had been receiving some extremely bleak polling numbers. As of today, Trump trails Hillary Clinton by 9 points in Virginia, by 8 points in Pennsylvania, by 6 points in Colorado, by 4 points in Florida, and by 3 points in North Carolina.

When we run all of these polls though our presidential forecasting model, it predicts that Trump has less than a 10 percent chance of winning the presidency.

Those are long odds. But they follow from the data. Here’s why our model is able to make such a strong prediction—and why we’re not the only forecasters to see the race this way.

Our model starts by forecasting the outcome of the presidential election in all 50 states and Washington, D.C., and then aggregates those results up to a national forecast. As expected, the polls show that there are a range of states that are “safe” for either Clinton or Trump—that is, where one candidate has at least a 99 percent chance of winning. But given our uncertainty about what could happen between now and Election Day, there also are states like Nevada, Ohio, and Iowa that could go either way. The full set of probabilities that Clinton or Trump will win each state are in the left sidebar on our presidential election overview page.

The next step is to convert all of these state probabilities into an overall chance that Clinton or Trump will win the election. For the sake of illustration, the simplest way to do this is to randomly simulate each state’s election outcome a large number of times and record the winner. From our current estimates, Clinton would win Nevada in 63 percent of simulations, Ohio in 46 percent of simulations, and so on. Again for ease, assume that the state outcomes are independent, so that whether Clinton wins Nevada has no bearing on whether she also wins Ohio. This isn’t completely realistic—and in fact, it’s not how our model works—but it’s a sufficient approximation. In each simulation, the candidate who wins each state receives all of that state’s electoral votes, which we add across all 50 states and D.C.

If we follow this procedure with our current set of state probabilities, Clinton comes out ahead in 99 percent of simulations. That is, in only 1 out of every 100 simulated elections does Donald Trump receive 270 or more electoral votes, and win the election. Clinton’s lead is so substantial that if we count up the electoral votes in the states she’s most likely to win, she gets to 273 by winning Colorado—an outcome that our model estimates is 94 percent likely.

On the other hand, finding a permutation of states that is consistent with the polling data and that gets Trump to 270 electoral votes is extremely difficult. In his most favorable scenario, Trump would have to win Colorado, where he only has a 6 percent chance, and Florida, where has has a 20 percent chance, and North Carolina, where he has a 35 percent chance, and Nevada, where he has a 37 percent chance, and every other state where his level of support is higher. If Trump loses any single one of these states, Clinton wins the election.

The other major forecasting models aren’t any more favorable to Trump’s chances. If we take the probabilities of winning each state currently being forecasted by The Upshot, FiveThirtyEight, The Huffington Post, PredictWise, and the Princeton Election Consortium, and run them through the same simulation, the result is nearly identical: Clinton’s implied chances of winning the national election are close to 100 percent:

• FiveThirtyEight: 98 percent
• The Upshot: 97 percent
• The Huffington Post: 99 percent
• Princeton Election Consortium: 98 percent
• PredictWise: 99 percent

The distributions of simulated electoral votes for Hillary Clinton under each model—again, by simply taking the state forecasts at face value—reinforce the challenge Trump faces. In every one of the models’ electoral vote histograms, there are almost no outcomes to the left of the blue line at 269 electoral votes, which is what Trump would need to win.

These histograms—and the chances of Clinton winning—are different from what each model is actually reporting as their national-level forecast because, like us, none of the other forecasters assume that state election outcomes are independent. If the polls are wrong, or if there’s a national swing in voter preferences toward Trump, then his odds should increase in many states at once: Nevada, Ohio, Florida, and so forth.

This adds extra uncertainty to the forecast, which widens the plausible range of electoral vote outcomes, and lowers Clinton’s chances of winning. The additional assumptions of The Upshot model, for example, bring Clinton’s overall chances down to 87 percent. In the FiveThirtyEight model, Clinton’s chances drop to 84 percent; and their histogram in particular looks very different than what I plotted above. (The Upshot recently published a pair of articles that explored these modeling choices more thoroughly.)

What this demonstrates, though, is that at this point in the campaign, the disagreements between the presidential models’ forecasts are primarily due to differences in the way uncertainty is carried through from the state forecasts to the national forecast. It is not that any of the forecasting models have a fundamentally more pro-Trump interpretation of the data. The models are essentially in agreement. Donald Trump is extremely unlikely to win the presidential election.

Save

Save