This is an archived article that was published on sltrib.com in 2016, and information in the article may be outdated. It is provided only for personal research purposes and may not be reprinted.
When the Salt Lake Tribune released its first pre-election poll of the 2016 cycle, it left my colleagues and me scratching our heads. Apparently the Owens campaign had a similar reaction, as reported in the Tribune ("Doug Owens holds edge over Rep. Mia Love in Tribune-Hinckley poll," June 10).
Generally, I chalk anomalous poll results up to run-of-the-mill statistical variation. This comes with the territory when doing random sampling; every poll has a one-in-20 chance of falling outside of the calculated margin of error. But even a casual review of the details of this poll reveal fundamental methodological flaws beyond what random chance accounts for.
The Tribune poll's single biggest red flag is the sample's age breakdown, which suggests a much older electorate than any in at least the past 30 years. Voters between the ages of 18 and 34 represent only 8 percent of the Tribune poll's sample. That is unprecedentedly low for this congressional district and for Utah overall. In fact – according to the publicly-available Utah Colleges Exit Poll – in 2014, 19 percent of voters belonged to this age group. In presidential years, younger voters make up an even larger share of voters – in 2012, it was 30 percent. An accurate proportion of younger voters would have been at least twice, and perhaps three times, as high as the published results.
This matters because both historically and by the pollster's own admission, Love's "support is young" and "[t]he younger the electorate, the stronger she will perform." In this poll, younger voters favor Rep. Love over Doug Owens 64 percent to 36 percent — a 28-percentage point advantage. Considering that dynamic, someone at the Tribune should have investigated why their sample did not have enough young voters to at least resemble previous election cycles.
While the age disparity is enough to cast suspicion on the results, there is another problem with this survey – it failed to sample enough strong partisans, especially Republicans. Party identification is the single best predictor of vote choice. Among voters sampled by the Tribune poll, only 13 percent self-identify as strong Republicans. The exit polling data from 2014 puts the percent of strong Republicans at nearly 26 percent of voters in this district, double what is represented in this survey.
The projected electorate asserted by this poll is far older and more Democratic than what is historically defensible, which artificially inflates support for Owens outside what most pollsters consider a reasonable estimate for this race. This is why both campaigns were reportedly surprised by the numbers in the Tribune poll. No one expected Doug Owens to be ahead in the polls, because in reality he is not.
These issues are certainly the result of relying mostly on robocall polling — conducted by SurveyUSA – for the research. Due to federal regulations, surveys administered using an autodialer cannot call cell phones, forcing researchers to rely solely on interviews over antiquated landline phones. As you might guess, this not only favors older voters, but also nearly wholly excludes voters under 34 years old, many of whom have not touched a landline at home since they were small children. In fact, over 50 percent of Utah residents live in homes that do not have a landline phone. SurveyUSA made a modest attempt to correct this bias using online panel interviews, but nine out of 10 of their survey interviews were done by a robocall over a landline phone.
This is not the first time the Tribune has published a poll afflicted by a suspect sample. In 2012, the newspaper released numbers that suggested Love would defeat Matheson by 12 percentage points. But after heavy criticism — including from the Matheson campaign research team — the newspaper retracted the poll. Matheson narrowly won re-election just days later. This research is similarly biased (albeit in the opposite direction).
Considering the recent ownership change, the Tribune has an opportunity to reassess its polling standards. Many publications, including the New York Times, ABC News and others, refuse to publish robocall surveys for the reasons outlined here. The Tribune should consider investing in and publishing higher quality polling so as not to distract the public dialogue with indefensible robocall polls. If the Tribune is unwilling to invest adequate resources and adopt high standards for its survey research, perhaps it's better off not sponsoring any polling of its own.
Scott Riding is the Managing Partner of Y2 Analytics, a public opinion research firm based in Salt Lake City, and is one of the pollsters working with the team to re-elect Rep. Mia Love.